Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251211 08:57:06.731086 13896 test_util.cc:276] Using random seed: 1414677452
2025-12-11T08:57:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-11T08:57:06Z Disabled control of system clock
I20251211 08:57:06.792088 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.62:39015
--webserver_interface=127.13.146.62
--webserver_port=0
--builtin_ntp_servers=127.13.146.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251211 08:57:07.064356 13905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:57:07.064931 13905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:57:07.065349 13905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:57:07.094635 13905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251211 08:57:07.094947 13905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:57:07.095224 13905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251211 08:57:07.095448 13905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251211 08:57:07.129017 13905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.62:39015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.146.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:57:07.130366 13905 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:57:07.131863 13905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:57:07.140841 13911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:07.142689 13912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:08.250722 13914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:08.252557 13913 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1106 milliseconds
I20251211 08:57:08.252671 13905 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251211 08:57:08.253741 13905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:57:08.256685 13905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:57:08.258105 13905 hybrid_clock.cc:648] HybridClock initialized: now 1765443428258063 us; error 60 us; skew 500 ppm
I20251211 08:57:08.258834 13905 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:57:08.264330 13905 webserver.cc:492] Webserver started at http://127.13.146.62:43581/ using document root <none> and password file <none>
I20251211 08:57:08.265133 13905 fs_manager.cc:362] Metadata directory not provided
I20251211 08:57:08.265331 13905 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:57:08.265782 13905 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:57:08.269791 13905 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data/instance:
uuid: "73702d54989f4959acfc4a6fdcede6ca"
format_stamp: "Formatted at 2025-12-11 08:57:08 on dist-test-slave-0dfs"
I20251211 08:57:08.270737 13905 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal/instance:
uuid: "73702d54989f4959acfc4a6fdcede6ca"
format_stamp: "Formatted at 2025-12-11 08:57:08 on dist-test-slave-0dfs"
I20251211 08:57:08.276849 13905 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251211 08:57:08.281885 13921 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:57:08.282683 13905 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251211 08:57:08.282969 13905 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal
uuid: "73702d54989f4959acfc4a6fdcede6ca"
format_stamp: "Formatted at 2025-12-11 08:57:08 on dist-test-slave-0dfs"
I20251211 08:57:08.283236 13905 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:57:08.344252 13905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:57:08.345465 13905 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:57:08.345829 13905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:57:08.408826 13905 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.62:39015
I20251211 08:57:08.408888 13972 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.62:39015 every 8 connection(s)
I20251211 08:57:08.411298 13905 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data/info.pb
I20251211 08:57:08.414566 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 13905
I20251211 08:57:08.415077 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal/instance
I20251211 08:57:08.417665 13973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:08.420387 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.61:34977
--webserver_interface=127.13.146.61
--webserver_port=0
--builtin_ntp_servers=127.13.146.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251211 08:57:08.429769 13973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:57:08.453835 13973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } has no permanent_uuid. Determining permanent_uuid...
W20251211 08:57:08.455976 13925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.61:34977: connect: Connection refused (error 111)
W20251211 08:57:08.459761 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:34977: Network error: Client connection negotiation failed: client connection to 127.13.146.61:34977: connect: Connection refused (error 111)
I20251211 08:57:08.509241 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } attempt: 1
W20251211 08:57:08.513411 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:34977: Network error: Client connection negotiation failed: client connection to 127.13.146.61:34977: connect: Connection refused (error 111)
I20251211 08:57:08.581784 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } attempt: 2
W20251211 08:57:08.585455 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:34977: Network error: Client connection negotiation failed: client connection to 127.13.146.61:34977: connect: Connection refused (error 111)
I20251211 08:57:08.676805 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } attempt: 3
W20251211 08:57:08.680583 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:34977: Network error: Client connection negotiation failed: client connection to 127.13.146.61:34977: connect: Connection refused (error 111)
W20251211 08:57:08.700094 13975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:57:08.700644 13975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:57:08.701063 13975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:57:08.729336 13975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251211 08:57:08.729627 13975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:57:08.729875 13975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251211 08:57:08.730123 13975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251211 08:57:08.763288 13975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.61:34977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.146.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:57:08.764609 13975 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:57:08.766161 13975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:57:08.776366 13984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:57:08.824005 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } attempt: 4
W20251211 08:57:08.831804 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:34977: Network error: Client connection negotiation failed: client connection to 127.13.146.61:34977: connect: Connection refused (error 111)
I20251211 08:57:09.131392 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } attempt: 5
W20251211 08:57:09.139138 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:34977: Network error: Client connection negotiation failed: client connection to 127.13.146.61:34977: connect: Connection refused (error 111)
I20251211 08:57:09.686728 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } attempt: 6
W20251211 08:57:09.712473 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:34977: Network error: Client connection negotiation failed: client connection to 127.13.146.61:34977: connect: Connection refused (error 111)
W20251211 08:57:08.777331 13985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:10.176927 13986 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1395 milliseconds
W20251211 08:57:10.177843 13975 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.401s user 0.414s sys 0.979s
W20251211 08:57:10.178143 13987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:10.178260 13975 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.402s user 0.414s sys 0.979s
I20251211 08:57:10.178541 13975 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251211 08:57:10.179684 13975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:57:10.182156 13975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:57:10.183482 13975 hybrid_clock.cc:648] HybridClock initialized: now 1765443430183454 us; error 35 us; skew 500 ppm
I20251211 08:57:10.184202 13975 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:57:10.191735 13975 webserver.cc:492] Webserver started at http://127.13.146.61:35969/ using document root <none> and password file <none>
I20251211 08:57:10.192736 13975 fs_manager.cc:362] Metadata directory not provided
I20251211 08:57:10.192924 13975 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:57:10.193310 13975 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:57:10.197351 13975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/data/instance:
uuid: "a4c9cd45be3947d09423ed4508ca2e91"
format_stamp: "Formatted at 2025-12-11 08:57:10 on dist-test-slave-0dfs"
I20251211 08:57:10.198307 13975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/wal/instance:
uuid: "a4c9cd45be3947d09423ed4508ca2e91"
format_stamp: "Formatted at 2025-12-11 08:57:10 on dist-test-slave-0dfs"
I20251211 08:57:10.204950 13975 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20251211 08:57:10.210227 13996 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:57:10.211316 13975 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20251211 08:57:10.211567 13975 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/wal
uuid: "a4c9cd45be3947d09423ed4508ca2e91"
format_stamp: "Formatted at 2025-12-11 08:57:10 on dist-test-slave-0dfs"
I20251211 08:57:10.211833 13975 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:57:10.269420 13975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:57:10.270746 13975 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:57:10.271152 13975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:57:10.333153 13975 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.61:34977
I20251211 08:57:10.333216 14047 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.61:34977 every 8 connection(s)
I20251211 08:57:10.335419 13975 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/data/info.pb
I20251211 08:57:10.340670 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 13975
I20251211 08:57:10.340809 14048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:10.341034 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-1/wal/instance
I20251211 08:57:10.343256 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.60:38671
--webserver_interface=127.13.146.60
--webserver_port=0
--builtin_ntp_servers=127.13.146.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251211 08:57:10.352923 14048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:57:10.373751 14048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:57:10.385669 14048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } has no permanent_uuid. Determining permanent_uuid...
W20251211 08:57:10.387769 13999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
W20251211 08:57:10.391678 14048 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
I20251211 08:57:10.441164 14048 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 1
W20251211 08:57:10.445446 14048 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
I20251211 08:57:10.513792 14048 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 2
W20251211 08:57:10.517688 14048 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
I20251211 08:57:10.609042 14048 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 3
W20251211 08:57:10.612996 14048 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
W20251211 08:57:10.637524 14050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:57:10.638062 14050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:57:10.638484 14050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:57:10.668361 14050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251211 08:57:10.668664 14050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:57:10.668889 14050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251211 08:57:10.669150 14050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251211 08:57:10.702402 14050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.60:38671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.146.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:57:10.703646 14050 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:57:10.705165 14050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:57:10.715860 14060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:57:10.756389 14048 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 4
W20251211 08:57:10.763142 14048 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
I20251211 08:57:10.772971 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } attempt: 7
I20251211 08:57:10.803580 13973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } has no permanent_uuid. Determining permanent_uuid...
W20251211 08:57:10.813419 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
I20251211 08:57:10.871973 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 1
W20251211 08:57:10.885525 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
I20251211 08:57:10.967056 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 2
W20251211 08:57:10.973201 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
I20251211 08:57:11.058686 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 3
I20251211 08:57:11.062620 14048 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 5
W20251211 08:57:11.069657 14048 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
W20251211 08:57:11.072186 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
I20251211 08:57:11.212721 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 4
W20251211 08:57:11.220080 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
I20251211 08:57:11.503636 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 5
W20251211 08:57:11.511333 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
I20251211 08:57:11.617209 14048 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 6
W20251211 08:57:11.654470 14048 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
W20251211 08:57:10.716214 14061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:12.006412 14062 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1286 milliseconds
W20251211 08:57:12.006685 14050 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.291s user 0.421s sys 0.862s
W20251211 08:57:12.007058 14050 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.292s user 0.422s sys 0.862s
W20251211 08:57:12.011698 14063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:57:12.011758 14050 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251211 08:57:12.013042 14050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:57:12.015652 14050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:57:12.016970 14050 hybrid_clock.cc:648] HybridClock initialized: now 1765443432016941 us; error 32 us; skew 500 ppm
I20251211 08:57:12.017724 14050 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:57:12.024546 14050 webserver.cc:492] Webserver started at http://127.13.146.60:39229/ using document root <none> and password file <none>
I20251211 08:57:12.025404 14050 fs_manager.cc:362] Metadata directory not provided
I20251211 08:57:12.025604 14050 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:57:12.026005 14050 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:57:12.031724 14050 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/data/instance:
uuid: "96e37aef4ef94e33a8a6807c7a9c404f"
format_stamp: "Formatted at 2025-12-11 08:57:12 on dist-test-slave-0dfs"
I20251211 08:57:12.032922 14050 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/wal/instance:
uuid: "96e37aef4ef94e33a8a6807c7a9c404f"
format_stamp: "Formatted at 2025-12-11 08:57:12 on dist-test-slave-0dfs"
I20251211 08:57:12.040397 14050 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20251211 08:57:12.046029 14075 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:57:12.047168 14050 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20251211 08:57:12.047506 14050 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/wal
uuid: "96e37aef4ef94e33a8a6807c7a9c404f"
format_stamp: "Formatted at 2025-12-11 08:57:12 on dist-test-slave-0dfs"
I20251211 08:57:12.047818 14050 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:57:12.063887 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 6
W20251211 08:57:12.071004 13973 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:38671: Network error: Client connection negotiation failed: client connection to 127.13.146.60:38671: connect: Connection refused (error 111)
I20251211 08:57:12.112424 14050 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:57:12.113842 14050 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:57:12.114233 14050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:57:12.179356 14050 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.60:38671
I20251211 08:57:12.179421 14128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.60:38671 every 8 connection(s)
I20251211 08:57:12.182024 14050 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/data/info.pb
I20251211 08:57:12.186570 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 14050
I20251211 08:57:12.187038 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-2/wal/instance
I20251211 08:57:12.187742 14129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:12.203176 14129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:57:12.225935 14129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:57:12.236586 14129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:57:12.254081 14129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f: Bootstrap starting.
I20251211 08:57:12.259523 14129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:12.261116 14129 log.cc:826] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f: Log is configured to *not* fsync() on all Append() calls
I20251211 08:57:12.265532 14129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f: No bootstrap required, opened a new log
I20251211 08:57:12.283452 14129 raft_consensus.cc:359] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:12.283950 14129 raft_consensus.cc:385] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:12.284111 14129 raft_consensus.cc:740] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e37aef4ef94e33a8a6807c7a9c404f, State: Initialized, Role: FOLLOWER
I20251211 08:57:12.284610 14129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:12.286535 14140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:12.287142 14140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:57:12.287890 14129 sys_catalog.cc:565] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: configured and running, proceeding with master startup.
W20251211 08:57:12.310096 14151 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:57:12.310535 14151 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:57:12.305455 14050 master_runner.cc:186] Error getting master registration for 127.13.146.62:39015: OK, instance_id { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" instance_seqno: 1765443428391961 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251211 08:57:12.326318 14050 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 08:57:12.368711 14083 tablet.cc:2378] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251211 08:57:12.630617 14140 raft_consensus.cc:493] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:12.631176 14140 raft_consensus.cc:515] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:12.633531 14140 leader_election.cc:290] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977)
I20251211 08:57:12.715010 14048 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 7
I20251211 08:57:12.734207 14048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Bootstrap starting.
I20251211 08:57:12.740948 14048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:12.743238 14048 log.cc:826] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Log is configured to *not* fsync() on all Append() calls
I20251211 08:57:12.748116 14048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: No bootstrap required, opened a new log
I20251211 08:57:12.773814 14048 raft_consensus.cc:359] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:12.774591 14048 raft_consensus.cc:385] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:12.774983 14048 raft_consensus.cc:740] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c9cd45be3947d09423ed4508ca2e91, State: Initialized, Role: FOLLOWER
I20251211 08:57:12.775885 14048 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:12.778997 14155 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:12.779744 14155 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:57:12.780910 14048 sys_catalog.cc:565] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: configured and running, proceeding with master startup.
I20251211 08:57:12.781108 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:12.781947 14023 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96e37aef4ef94e33a8a6807c7a9c404f in term 0.
I20251211 08:57:12.783191 14081 leader_election.cc:304] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91; no voters:
I20251211 08:57:12.784112 14140 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:12.784454 14140 raft_consensus.cc:493] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:12.784785 14140 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:12.791281 14140 raft_consensus.cc:515] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:12.793514 14140 leader_election.cc:290] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [CANDIDATE]: Term 1 election: Requested vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977)
I20251211 08:57:12.794112 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91"
I20251211 08:57:12.794628 14023 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:12.801517 14023 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96e37aef4ef94e33a8a6807c7a9c404f in term 1.
I20251211 08:57:12.802560 14081 leader_election.cc:304] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91; no voters:
I20251211 08:57:12.803339 14140 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:12.817071 14140 raft_consensus.cc:697] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 1 LEADER]: Becoming Leader. State: Replica: 96e37aef4ef94e33a8a6807c7a9c404f, State: Running, Role: LEADER
I20251211 08:57:12.818040 14140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:12.828760 14159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96e37aef4ef94e33a8a6807c7a9c404f. Latest consensus state: current_term: 1 leader_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:12.829509 14159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: This master's current role is: LEADER
I20251211 08:57:12.838904 14163 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251211 08:57:12.846160 14163 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251211 08:57:12.861833 14023 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Refusing update from remote peer 96e37aef4ef94e33a8a6807c7a9c404f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:12.863454 14159 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:12.898810 14155 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96e37aef4ef94e33a8a6807c7a9c404f. Latest consensus state: current_term: 1 leader_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:12.899502 14155 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:57:12.906201 14140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:12.906863 14140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: This master's current role is: LEADER
I20251211 08:57:12.912930 14163 catalog_manager.cc:1357] Generated new cluster ID: ad83da70c3aa44c788d5a566bdc1630f
I20251211 08:57:12.913177 14163 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251211 08:57:12.924145 14155 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:12.924716 14155 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:57:12.875447 13975 master_runner.cc:186] Error getting master registration for 127.13.146.62:39015: OK, instance_id { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" instance_seqno: 1765443428391961 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251211 08:57:12.926071 13975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 08:57:12.930871 14170 catalog_manager.cc:1568] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:57:12.931217 14170 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:57:12.965449 14163 catalog_manager.cc:1380] Generated new certificate authority record
I20251211 08:57:12.968391 14163 catalog_manager.cc:1514] Loading token signing keys...
I20251211 08:57:13.004923 14163 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f: Generated new TSK 0
I20251211 08:57:13.006100 14163 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251211 08:57:13.033939 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.146.1:0
--local_ip_for_outbound_sockets=127.13.146.1
--webserver_interface=127.13.146.1
--webserver_port=0
--tserver_master_addrs=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--builtin_ntp_servers=127.13.146.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251211 08:57:13.104436 13973 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } attempt: 7
I20251211 08:57:13.147697 13973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: Bootstrap starting.
I20251211 08:57:13.171584 13973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:13.183351 14080 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.13.146.62:39015 timed out after 0.549s (SENT)
I20251211 08:57:13.196655 13973 log.cc:826] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: Log is configured to *not* fsync() on all Append() calls
I20251211 08:57:13.235666 13973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: No bootstrap required, opened a new log
I20251211 08:57:13.402830 13973 raft_consensus.cc:359] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:13.405416 13973 raft_consensus.cc:385] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:13.406322 13973 raft_consensus.cc:740] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73702d54989f4959acfc4a6fdcede6ca, State: Initialized, Role: FOLLOWER
I20251211 08:57:13.428439 13973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:13.457901 13973 sys_catalog.cc:565] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: configured and running, proceeding with master startup.
I20251211 08:57:13.458065 13945 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:13.458824 13945 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96e37aef4ef94e33a8a6807c7a9c404f in term 0.
I20251211 08:57:13.459677 13944 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca"
I20251211 08:57:13.460247 13944 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:13.460487 14180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:13.462507 14180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:57:13.462980 13947 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca"
W20251211 08:57:13.463814 13947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48744 (request call id 3) took 670 ms (client timeout 632 ms). Trace:
W20251211 08:57:13.464133 13947 rpcz_store.cc:269] 1211 08:57:12.793211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:12.793357 (+ 146us) service_pool.cc:225] Handling call
1211 08:57:13.463737 (+670380us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:57:13.465026 13948 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
W20251211 08:57:13.465742 13948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48744 (request call id 2) took 831 ms (client timeout 549 ms). Trace:
W20251211 08:57:13.466019 13948 rpcz_store.cc:269] 1211 08:57:12.634042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:12.634193 (+ 151us) service_pool.cc:225] Handling call
1211 08:57:13.465663 (+831470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:57:13.471266 13944 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96e37aef4ef94e33a8a6807c7a9c404f in term 1.
I20251211 08:57:13.472234 13946 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Refusing update from remote peer 96e37aef4ef94e33a8a6807c7a9c404f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:13.479292 14140 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [LEADER]: Connected to new peer: Peer: permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:13.514575 14190 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:13.520243 14190 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:13.524137 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:13.523129 13946 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:13.526300 14190 leader_election.cc:290] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:13.527426 13999 leader_election.cc:304] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4c9cd45be3947d09423ed4508ca2e91; no voters: 73702d54989f4959acfc4a6fdcede6ca, 96e37aef4ef94e33a8a6807c7a9c404f
I20251211 08:57:13.529096 14190 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:13.558521 14180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96e37aef4ef94e33a8a6807c7a9c404f. Latest consensus state: current_term: 1 leader_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:13.559177 14180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:57:13.560928 14192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:13.561709 14192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:57:13.572693 14201 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20251211 08:57:13.605782 14176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:57:13.606365 14176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:57:13.607113 14176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:57:13.639338 14176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:57:13.640358 14176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.146.1
I20251211 08:57:13.673676 14176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.146.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.146.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.146.1
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:57:13.674996 14176 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:57:13.676405 14176 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:57:13.688730 14203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:57:13.910008 14140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:13.911173 14140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: This master's current role is: LEADER
I20251211 08:57:13.938431 14170 catalog_manager.cc:1269] Loaded cluster ID: ad83da70c3aa44c788d5a566bdc1630f
I20251211 08:57:13.938905 14170 catalog_manager.cc:1562] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: loading cluster ID for follower catalog manager: success
I20251211 08:57:13.948011 14170 catalog_manager.cc:1584] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: acquiring CA information for follower catalog manager: success
I20251211 08:57:13.965924 14170 catalog_manager.cc:1612] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251211 08:57:14.621502 14201 catalog_manager.cc:1269] Loaded cluster ID: ad83da70c3aa44c788d5a566bdc1630f
I20251211 08:57:14.621861 14201 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: loading cluster ID for follower catalog manager: success
I20251211 08:57:14.650525 14201 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: acquiring CA information for follower catalog manager: success
I20251211 08:57:14.655712 14201 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251211 08:57:15.283546 14212 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:15.285180 14212 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:15.288465 14212 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:15.290596 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:15.291826 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:15.293968 13925 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:57:15.294667 14212 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:15.363070 14213 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:15.373535 14213 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:15.377079 13944 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:15.384197 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:15.386054 13999 leader_election.cc:304] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4c9cd45be3947d09423ed4508ca2e91; no voters: 73702d54989f4959acfc4a6fdcede6ca, 96e37aef4ef94e33a8a6807c7a9c404f
I20251211 08:57:15.395124 14213 leader_election.cc:290] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:15.396423 14213 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:15.824929 14215 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:15.825404 14215 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:15.827895 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:15.830569 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:15.831966 13924 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:57:15.843405 14216 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:15.851725 14215 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:16.355618 14218 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:16.356369 14218 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:16.359655 13944 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:16.368433 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:16.370373 13999 leader_election.cc:304] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4c9cd45be3947d09423ed4508ca2e91; no voters: 73702d54989f4959acfc4a6fdcede6ca, 96e37aef4ef94e33a8a6807c7a9c404f
I20251211 08:57:16.379237 14218 leader_election.cc:290] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:16.380347 14218 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:16.895434 14221 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:16.896569 14221 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:16.900425 13944 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:16.903544 14221 leader_election.cc:290] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:16.906198 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:16.908141 13999 leader_election.cc:304] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4c9cd45be3947d09423ed4508ca2e91; no voters: 73702d54989f4959acfc4a6fdcede6ca, 96e37aef4ef94e33a8a6807c7a9c404f
I20251211 08:57:16.909068 14221 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251211 08:57:15.093902 14202 debug-util.cc:398] Leaking SignalData structure 0x7b0800034c40 after lost signal to thread 14176
W20251211 08:57:17.708755 14202 kernel_stack_watchdog.cc:198] Thread 14176 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20251211 08:57:17.709981 14176 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 4.019s user 1.417s sys 2.257s
W20251211 08:57:17.710531 14176 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 4.020s user 1.417s sys 2.257s
W20251211 08:57:13.690632 14204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:17.712245 14206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:17.715955 14205 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 4019 milliseconds
I20251211 08:57:17.716471 14176 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251211 08:57:17.717871 14176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:57:17.720436 14176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:57:17.721869 14176 hybrid_clock.cc:648] HybridClock initialized: now 1765443437721848 us; error 65 us; skew 500 ppm
I20251211 08:57:17.722926 14176 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:57:17.730482 14176 webserver.cc:492] Webserver started at http://127.13.146.1:33437/ using document root <none> and password file <none>
I20251211 08:57:17.731768 14176 fs_manager.cc:362] Metadata directory not provided
I20251211 08:57:17.732067 14176 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:57:17.732618 14176 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:57:17.738574 14176 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/data/instance:
uuid: "31404d5f58e548519239569b150d975b"
format_stamp: "Formatted at 2025-12-11 08:57:17 on dist-test-slave-0dfs"
I20251211 08:57:17.740041 14176 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/instance:
uuid: "31404d5f58e548519239569b150d975b"
format_stamp: "Formatted at 2025-12-11 08:57:17 on dist-test-slave-0dfs"
I20251211 08:57:17.751718 14226 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:17.752271 14226 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:17.753273 14176 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.008s sys 0.005s
I20251211 08:57:17.755080 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:17.758209 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:17.759907 13924 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:57:17.765167 14226 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:17.766330 14226 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:17.772759 14228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:57:17.773928 14176 fs_manager.cc:730] Time spent opening block manager: real 0.016s user 0.004s sys 0.013s
I20251211 08:57:17.774235 14176 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal
uuid: "31404d5f58e548519239569b150d975b"
format_stamp: "Formatted at 2025-12-11 08:57:17 on dist-test-slave-0dfs"
I20251211 08:57:17.774557 14176 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:57:17.848599 14176 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:57:17.850500 14176 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:57:17.851004 14176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:57:17.854080 14176 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251211 08:57:17.858721 14176 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251211 08:57:17.858938 14176 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251211 08:57:17.859170 14176 ts_tablet_manager.cc:616] Registered 0 tablets
I20251211 08:57:17.859328 14176 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251211 08:57:17.938367 14264 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:17.940455 14264 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:17.953644 13944 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:17.957935 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:17.959858 13999 leader_election.cc:304] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4c9cd45be3947d09423ed4508ca2e91; no voters: 73702d54989f4959acfc4a6fdcede6ca, 96e37aef4ef94e33a8a6807c7a9c404f
I20251211 08:57:17.969811 14264 leader_election.cc:290] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:17.971100 14264 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:18.217080 14176 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.1:36181
I20251211 08:57:18.217227 14343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.1:36181 every 8 connection(s)
I20251211 08:57:18.220395 14176 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/data/info.pb
I20251211 08:57:18.221733 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 14176
I20251211 08:57:18.222316 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/instance
I20251211 08:57:18.246599 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.146.2:0
--local_ip_for_outbound_sockets=127.13.146.2
--webserver_interface=127.13.146.2
--webserver_port=0
--tserver_master_addrs=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--builtin_ntp_servers=127.13.146.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251211 08:57:18.354276 14352 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:18.354750 14352 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:18.360206 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:18.360652 14352 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:18.362228 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:18.365023 13924 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:57:18.365759 14352 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:18.369395 14344 heartbeater.cc:344] Connected to a master server at 127.13.146.61:34977
I20251211 08:57:18.370059 14344 heartbeater.cc:461] Registering TS with master...
I20251211 08:57:18.371928 14344 heartbeater.cc:507] Master 127.13.146.61:34977 requested a full tablet report, sending...
I20251211 08:57:18.376233 14013 ts_manager.cc:194] Registered new tserver with Master: 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:18.377113 14345 heartbeater.cc:344] Connected to a master server at 127.13.146.60:38671
I20251211 08:57:18.377955 14345 heartbeater.cc:461] Registering TS with master...
I20251211 08:57:18.378852 14345 heartbeater.cc:507] Master 127.13.146.60:38671 requested a full tablet report, sending...
I20251211 08:57:18.381465 14094 ts_manager.cc:194] Registered new tserver with Master: 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:18.384037 14094 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.146.1:45533
I20251211 08:57:18.400209 14349 heartbeater.cc:344] Connected to a master server at 127.13.146.62:39015
I20251211 08:57:18.400609 14349 heartbeater.cc:461] Registering TS with master...
I20251211 08:57:18.401314 14349 heartbeater.cc:507] Master 127.13.146.62:39015 requested a full tablet report, sending...
I20251211 08:57:18.404218 13938 ts_manager.cc:194] Registered new tserver with Master: 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:18.534755 14355 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:18.535287 14355 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:18.537127 14355 leader_election.cc:290] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:18.538245 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:18.538288 13944 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:18.539892 13999 leader_election.cc:304] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4c9cd45be3947d09423ed4508ca2e91; no voters: 73702d54989f4959acfc4a6fdcede6ca, 96e37aef4ef94e33a8a6807c7a9c404f
I20251211 08:57:18.540616 14355 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251211 08:57:18.692075 14348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:57:18.692893 14348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:57:18.693989 14348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:57:18.725003 14348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:57:18.725832 14348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.146.2
I20251211 08:57:18.758296 14348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.146.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.146.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.146.2
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:57:18.759626 14348 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:57:18.761184 14348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:57:18.772441 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:57:18.887804 14345 heartbeater.cc:499] Master 127.13.146.60:38671 was elected leader, sending a full tablet report...
I20251211 08:57:18.983593 14368 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:18.984496 14368 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:18.988445 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:18.993150 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:18.995265 13924 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:57:19.000587 14368 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:19.001734 14368 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:19.164057 14370 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:19.165154 14370 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:19.169129 13944 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:19.172993 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:19.180272 13999 leader_election.cc:304] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4c9cd45be3947d09423ed4508ca2e91; no voters: 73702d54989f4959acfc4a6fdcede6ca, 96e37aef4ef94e33a8a6807c7a9c404f
I20251211 08:57:19.197279 14370 leader_election.cc:290] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:19.198892 14370 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:19.564079 14372 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:19.564968 14372 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:19.568396 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:19.571939 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:19.573725 13924 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:57:19.576705 14372 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:19.576838 14373 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:20.134660 14375 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:20.135537 14375 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:20.138286 14375 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:20.140630 14376 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:20.141332 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:20.141513 14376 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:20.144129 14376 leader_election.cc:290] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:20.145161 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:20.145793 14023 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73702d54989f4959acfc4a6fdcede6ca in term 1.
I20251211 08:57:20.148905 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:20.150311 13925 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca, a4c9cd45be3947d09423ed4508ca2e91; no voters: 96e37aef4ef94e33a8a6807c7a9c404f
I20251211 08:57:20.151526 14375 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:57:20.151917 14375 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Starting leader election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:20.152314 14375 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:57:20.156786 13944 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:20.159749 13999 leader_election.cc:304] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4c9cd45be3947d09423ed4508ca2e91; no voters: 73702d54989f4959acfc4a6fdcede6ca, 96e37aef4ef94e33a8a6807c7a9c404f
I20251211 08:57:20.162087 14376 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:20.164562 14375 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:20.167537 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91"
I20251211 08:57:20.167912 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f"
I20251211 08:57:20.169355 14375 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 election: Requested vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:20.169370 13924 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:57:20.170171 14375 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251211 08:57:18.773818 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:20.428476 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:20.175980 14361 debug-util.cc:398] Leaking SignalData structure 0x7b0800034c40 after lost signal to thread 14348
W20251211 08:57:20.429105 14364 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1654 milliseconds
W20251211 08:57:20.426128 14348 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.652s user 0.556s sys 1.050s
W20251211 08:57:20.429811 14348 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.656s user 0.556s sys 1.050s
I20251211 08:57:20.430087 14348 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251211 08:57:20.433511 14348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:57:20.435817 14348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:57:20.437248 14348 hybrid_clock.cc:648] HybridClock initialized: now 1765443440437184 us; error 54 us; skew 500 ppm
I20251211 08:57:20.438364 14348 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:57:20.445652 14348 webserver.cc:492] Webserver started at http://127.13.146.2:32973/ using document root <none> and password file <none>
I20251211 08:57:20.446904 14348 fs_manager.cc:362] Metadata directory not provided
I20251211 08:57:20.447175 14348 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:57:20.447813 14348 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:57:20.453722 14348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/data/instance:
uuid: "b276f77f87254a5d909c5ab940a39256"
format_stamp: "Formatted at 2025-12-11 08:57:20 on dist-test-slave-0dfs"
I20251211 08:57:20.454789 14348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/instance:
uuid: "b276f77f87254a5d909c5ab940a39256"
format_stamp: "Formatted at 2025-12-11 08:57:20 on dist-test-slave-0dfs"
I20251211 08:57:20.462108 14348 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20251211 08:57:20.467446 14381 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:57:20.468377 14348 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20251211 08:57:20.468673 14348 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal
uuid: "b276f77f87254a5d909c5ab940a39256"
format_stamp: "Formatted at 2025-12-11 08:57:20 on dist-test-slave-0dfs"
I20251211 08:57:20.468943 14348 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:57:20.517534 14348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:57:20.518812 14348 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:57:20.519178 14348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:57:20.521661 14348 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251211 08:57:20.525997 14348 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251211 08:57:20.526171 14348 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251211 08:57:20.526343 14348 ts_tablet_manager.cc:616] Registered 0 tablets
I20251211 08:57:20.526459 14348 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251211 08:57:20.654987 14348 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.2:36153
I20251211 08:57:20.655040 14494 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.2:36153 every 8 connection(s)
I20251211 08:57:20.657311 14348 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/data/info.pb
I20251211 08:57:20.665696 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 14348
I20251211 08:57:20.666220 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/instance
I20251211 08:57:20.693818 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.146.3:0
--local_ip_for_outbound_sockets=127.13.146.3
--webserver_interface=127.13.146.3
--webserver_port=0
--tserver_master_addrs=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--builtin_ntp_servers=127.13.146.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251211 08:57:20.706283 14506 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:20.706751 14506 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:20.709242 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:20.710850 14496 heartbeater.cc:344] Connected to a master server at 127.13.146.60:38671
I20251211 08:57:20.711354 14496 heartbeater.cc:461] Registering TS with master...
I20251211 08:57:20.712236 14506 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:20.712527 14496 heartbeater.cc:507] Master 127.13.146.60:38671 requested a full tablet report, sending...
I20251211 08:57:20.712610 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:20.714336 13924 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:57:20.714864 14094 ts_manager.cc:194] Registered new tserver with Master: b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:20.715523 14495 heartbeater.cc:344] Connected to a master server at 127.13.146.61:34977
I20251211 08:57:20.715585 14506 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251211 08:57:20.715835 14495 heartbeater.cc:461] Registering TS with master...
I20251211 08:57:20.716468 14094 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.146.2:33483
I20251211 08:57:20.716662 14495 heartbeater.cc:507] Master 127.13.146.61:34977 requested a full tablet report, sending...
I20251211 08:57:20.719029 14013 ts_manager.cc:194] Registered new tserver with Master: b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:20.722419 14497 heartbeater.cc:344] Connected to a master server at 127.13.146.62:39015
I20251211 08:57:20.722731 14497 heartbeater.cc:461] Registering TS with master...
I20251211 08:57:20.723215 14497 heartbeater.cc:507] Master 127.13.146.62:39015 requested a full tablet report, sending...
I20251211 08:57:20.724911 13938 ts_manager.cc:194] Registered new tserver with Master: b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:20.736706 13944 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Rejecting Update request from peer 96e37aef4ef94e33a8a6807c7a9c404f for earlier term 1. Current term is 2. Ops: []
I20251211 08:57:20.737746 14374 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251211 08:57:20.739435 14507 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 1 LEADER]: Stepping down as leader of term 1
I20251211 08:57:20.739696 14507 raft_consensus.cc:740] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96e37aef4ef94e33a8a6807c7a9c404f, State: Running, Role: LEADER
I20251211 08:57:20.740352 14507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:20.741685 14507 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 1 FOLLOWER]: Advancing to term 2
W20251211 08:57:21.008075 14505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:57:21.008486 14505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:57:21.008929 14505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:57:21.038229 14505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:57:21.039096 14505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.146.3
I20251211 08:57:21.072775 14505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.146.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.146.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.146.3
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:57:21.074015 14505 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:57:21.075469 14505 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:57:21.088099 14513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:57:21.160984 14519 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:21.162078 14519 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:21.165767 13944 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:21.166976 13944 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4c9cd45be3947d09423ed4508ca2e91 in current term 2: Already voted for candidate 73702d54989f4959acfc4a6fdcede6ca in this term.
I20251211 08:57:21.170449 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:21.171310 14104 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4c9cd45be3947d09423ed4508ca2e91 in term 2.
I20251211 08:57:21.173231 13999 leader_election.cc:304] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91; no voters: 73702d54989f4959acfc4a6fdcede6ca
I20251211 08:57:21.175874 14519 leader_election.cc:290] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:21.177006 14519 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:57:21.177994 14519 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 96e37aef4ef94e33a8a6807c7a9c404f)
I20251211 08:57:21.178709 14519 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:57:21.192678 14519 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:21.196120 13944 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca"
I20251211 08:57:21.196785 13944 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a4c9cd45be3947d09423ed4508ca2e91 in current term 2: Already voted for candidate 73702d54989f4959acfc4a6fdcede6ca in this term.
I20251211 08:57:21.199314 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c9cd45be3947d09423ed4508ca2e91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f"
I20251211 08:57:21.208401 14104 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4c9cd45be3947d09423ed4508ca2e91 in term 2.
I20251211 08:57:21.208781 14519 leader_election.cc:290] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 election: Requested vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:21.212116 13999 leader_election.cc:304] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91; no voters: 73702d54989f4959acfc4a6fdcede6ca
I20251211 08:57:21.213826 14519 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 2 FOLLOWER]: Leader election won for term 2
I20251211 08:57:21.215072 14519 raft_consensus.cc:697] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [term 2 LEADER]: Becoming Leader. State: Replica: a4c9cd45be3947d09423ed4508ca2e91, State: Running, Role: LEADER
I20251211 08:57:21.216395 14519 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:21.220417 14496 heartbeater.cc:499] Master 127.13.146.60:38671 was elected leader, sending a full tablet report...
I20251211 08:57:21.241657 14520 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c9cd45be3947d09423ed4508ca2e91. Latest consensus state: current_term: 2 leader_uuid: "a4c9cd45be3947d09423ed4508ca2e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:21.242609 14520 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: This master's current role is: LEADER
I20251211 08:57:21.252807 14522 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251211 08:57:21.263273 14522 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251211 08:57:21.268231 14522 catalog_manager.cc:1269] Loaded cluster ID: ad83da70c3aa44c788d5a566bdc1630f
I20251211 08:57:21.268620 14522 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251211 08:57:21.275986 14522 catalog_manager.cc:1514] Loading token signing keys...
I20251211 08:57:21.279922 14522 catalog_manager.cc:6038] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Loaded TSK: 0
I20251211 08:57:21.281876 14522 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251211 08:57:21.324594 14151 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251211 08:57:21.693123 13944 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Refusing update from remote peer a4c9cd45be3947d09423ed4508ca2e91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251211 08:57:21.695575 14520 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251211 08:57:21.721324 14523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c9cd45be3947d09423ed4508ca2e91. Latest consensus state: current_term: 2 leader_uuid: "a4c9cd45be3947d09423ed4508ca2e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:21.722540 14523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:57:21.770347 14104 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 2 FOLLOWER]: Refusing update from remote peer a4c9cd45be3947d09423ed4508ca2e91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251211 08:57:21.773064 14519 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251211 08:57:21.781903 14523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a4c9cd45be3947d09423ed4508ca2e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:21.783424 14523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:57:21.788283 14519 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a4c9cd45be3947d09423ed4508ca2e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:21.790042 14519 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: This master's current role is: LEADER
W20251211 08:57:21.801834 14491 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad060 after lost signal to thread 14356
W20251211 08:57:21.803318 14491 debug-util.cc:398] Leaking SignalData structure 0x7b08000adf60 after lost signal to thread 14494
I20251211 08:57:21.824255 14527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c9cd45be3947d09423ed4508ca2e91. Latest consensus state: current_term: 2 leader_uuid: "a4c9cd45be3947d09423ed4508ca2e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:21.825291 14527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:57:21.830907 14531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a4c9cd45be3947d09423ed4508ca2e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:21.832304 14531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:57:21.905961 14344 heartbeater.cc:499] Master 127.13.146.61:34977 was elected leader, sending a full tablet report...
I20251211 08:57:22.290431 14495 heartbeater.cc:499] Master 127.13.146.61:34977 was elected leader, sending a full tablet report...
I20251211 08:57:22.296149 14531 raft_consensus.cc:493] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4c9cd45be3947d09423ed4508ca2e91)
I20251211 08:57:22.296866 14531 raft_consensus.cc:515] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:22.301478 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:22.304796 13946 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:22.307626 14081 leader_election.cc:304] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96e37aef4ef94e33a8a6807c7a9c404f; no voters: 73702d54989f4959acfc4a6fdcede6ca, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:57:22.308408 14527 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251211 08:57:22.308988 14531 leader_election.cc:290] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977)
I20251211 08:57:22.370834 14532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a4c9cd45be3947d09423ed4508ca2e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:22.371564 14532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: This master's current role is: LEADER
I20251211 08:57:22.690984 14535 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4c9cd45be3947d09423ed4508ca2e91)
I20251211 08:57:22.692588 14535 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:22.697122 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:22.697561 14535 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:57:22.699121 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:57:22.702069 13925 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:57:22.703239 14535 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251211 08:57:22.489286 14512 debug-util.cc:398] Leaking SignalData structure 0x7b0800034c40 after lost signal to thread 14505
W20251211 08:57:22.869788 14512 kernel_stack_watchdog.cc:198] Thread 14505 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20251211 08:57:21.088340 14514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:22.873047 14516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:57:22.873643 14505 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.786s user 0.595s sys 0.930s
W20251211 08:57:22.874074 14505 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.787s user 0.595s sys 0.931s
W20251211 08:57:22.876364 14515 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1787 milliseconds
I20251211 08:57:22.876381 14505 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251211 08:57:22.877521 14505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:57:22.879887 14505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:57:22.881476 14505 hybrid_clock.cc:648] HybridClock initialized: now 1765443442881398 us; error 87 us; skew 500 ppm
I20251211 08:57:22.882567 14505 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:57:22.888341 14505 webserver.cc:492] Webserver started at http://127.13.146.3:40939/ using document root <none> and password file <none>
I20251211 08:57:22.889276 14505 fs_manager.cc:362] Metadata directory not provided
I20251211 08:57:22.889467 14505 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:57:22.889918 14505 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:57:22.894054 14505 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/data/instance:
uuid: "c5e624bcfd154cd586ef1dedb9940063"
format_stamp: "Formatted at 2025-12-11 08:57:22 on dist-test-slave-0dfs"
I20251211 08:57:22.895112 14505 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/instance:
uuid: "c5e624bcfd154cd586ef1dedb9940063"
format_stamp: "Formatted at 2025-12-11 08:57:22 on dist-test-slave-0dfs"
I20251211 08:57:22.902405 14505 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20251211 08:57:22.908804 14540 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:57:22.909808 14505 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20251211 08:57:22.910080 14505 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal
uuid: "c5e624bcfd154cd586ef1dedb9940063"
format_stamp: "Formatted at 2025-12-11 08:57:22 on dist-test-slave-0dfs"
I20251211 08:57:22.910352 14505 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:57:22.957819 14505 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:57:22.959314 14505 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:57:22.959704 14505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:57:22.962231 14505 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251211 08:57:22.966205 14505 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251211 08:57:22.966456 14505 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251211 08:57:22.966687 14505 ts_tablet_manager.cc:616] Registered 0 tablets
I20251211 08:57:22.966858 14505 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251211 08:57:23.137636 14505 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.3:37291
I20251211 08:57:23.137797 14652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.3:37291 every 8 connection(s)
I20251211 08:57:23.140244 14505 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/data/info.pb
I20251211 08:57:23.150875 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 14505
I20251211 08:57:23.151479 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/instance
I20251211 08:57:23.195127 14653 heartbeater.cc:344] Connected to a master server at 127.13.146.61:34977
I20251211 08:57:23.195588 14653 heartbeater.cc:461] Registering TS with master...
I20251211 08:57:23.197094 14653 heartbeater.cc:507] Master 127.13.146.61:34977 requested a full tablet report, sending...
I20251211 08:57:23.201294 14013 ts_manager.cc:194] Registered new tserver with Master: c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:23.203936 14013 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.146.3:43327
I20251211 08:57:23.217628 14655 heartbeater.cc:344] Connected to a master server at 127.13.146.62:39015
I20251211 08:57:23.218010 14655 heartbeater.cc:461] Registering TS with master...
I20251211 08:57:23.218683 14655 heartbeater.cc:507] Master 127.13.146.62:39015 requested a full tablet report, sending...
I20251211 08:57:23.221064 13938 ts_manager.cc:194] Registered new tserver with Master: c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:23.221655 14654 heartbeater.cc:344] Connected to a master server at 127.13.146.60:38671
I20251211 08:57:23.221933 14654 heartbeater.cc:461] Registering TS with master...
I20251211 08:57:23.222432 14654 heartbeater.cc:507] Master 127.13.146.60:38671 requested a full tablet report, sending...
I20251211 08:57:23.224397 14094 ts_manager.cc:194] Registered new tserver with Master: c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:23.232146 13896 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251211 08:57:23.298648 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_9be8f0bdb87d4f3bb4535d264277bfa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:23.302050 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9be8f0bdb87d4f3bb4535d264277bfa1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:23.302151 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_79f3bca103914228ae40b42bf4fcfb18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:23.302731 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f11b89dd7d3d438f91f520a65605969a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:23.304000 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79f3bca103914228ae40b42bf4fcfb18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:23.304438 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f11b89dd7d3d438f91f520a65605969a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:23.305562 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ed5bf9b8a1c94f46ba4e5184aef6b38d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:23.307355 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed5bf9b8a1c94f46ba4e5184aef6b38d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:23.464349 14011 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f11b89dd7d3d438f91f520a65605969a" } new_table_name: "default.Table_2114d5bcfd134d01b68352cbe97e506f" modify_external_catalogs: true
I20251211 08:57:23.469437 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_79f3bca103914228ae40b42bf4fcfb18" } modify_external_catalogs: true
I20251211 08:57:23.470165 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_79f3bca103914228ae40b42bf4fcfb18" } modify_external_catalogs: true
I20251211 08:57:23.474449 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ed5bf9b8a1c94f46ba4e5184aef6b38d" } new_table_name: "default.Table_8518029aae6a499fbca00760d422677b" modify_external_catalogs: true
I20251211 08:57:23.495187 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9be8f0bdb87d4f3bb4535d264277bfa1" } modify_external_catalogs: true
I20251211 08:57:23.495750 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9be8f0bdb87d4f3bb4535d264277bfa1" } modify_external_catalogs: true
W20251211 08:57:23.532054 14011 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19b6f3954b294c49ba0688174aa87f5d has no consensus state
I20251211 08:57:23.560928 14013 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet a304bda4c18f4a65b177fc55a55b3c49
W20251211 08:57:23.567875 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e268fd8444ed4974bd6c7b4c5b7c63ce consensus state has no leader
W20251211 08:57:23.601015 13997 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 19b6f3954b294c49ba0688174aa87f5d (table default.Table_2114d5bcfd134d01b68352cbe97e506f [id=35601cbbebb14fdda09e3c3d5cc47bd7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 19b6f3954b294c49ba0688174aa87f5d has no consensus state
I20251211 08:57:23.626116 14588 tablet_service.cc:1505] Processing CreateTablet for tablet e268fd8444ed4974bd6c7b4c5b7c63ce (DEFAULT_TABLE table=default.Table_ed5bf9b8a1c94f46ba4e5184aef6b38d [id=d25467f799644415bf42e502694acce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:23.628279 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e268fd8444ed4974bd6c7b4c5b7c63ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:23.636564 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.62:39015, user_credentials={real_user=slave}} blocked reactor thread for 90829.3us
I20251211 08:57:23.673885 14010 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 3c0899f2f77c475f91656f44b93f6a4b
I20251211 08:57:23.693455 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_1ceb0e35a939466db7ce05224bf6caaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:23.697286 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ceb0e35a939466db7ce05224bf6caaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:23.702193 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f8b90871ac1a41f6aa1540128ed52976"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:23.704233 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8b90871ac1a41f6aa1540128ed52976 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:23.709017 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_534b28534eba477e8364a8648fba0c60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:23.709323 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b4370c3e38fa4981a2782889fe923f01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:23.710491 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_534b28534eba477e8364a8648fba0c60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:23.710938 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4370c3e38fa4981a2782889fe923f01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:23.712929 14653 heartbeater.cc:499] Master 127.13.146.61:34977 was elected leader, sending a full tablet report...
I20251211 08:57:23.715478 13896 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu with pid 13905
I20251211 08:57:23.736920 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8518029aae6a499fbca00760d422677b" } new_table_name: "default.Table_5232378b2b1448838dc4b77baf176467" modify_external_catalogs: true
I20251211 08:57:23.761605 14703 tablet_bootstrap.cc:492] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:23.768299 14703 tablet_bootstrap.cc:654] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:23.770740 14703 log.cc:826] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063: Log is configured to *not* fsync() on all Append() calls
I20251211 08:57:23.776571 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_2114d5bcfd134d01b68352cbe97e506f" } modify_external_catalogs: true
I20251211 08:57:23.777351 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_2114d5bcfd134d01b68352cbe97e506f" } modify_external_catalogs: true
I20251211 08:57:23.781551 14703 tablet_bootstrap.cc:492] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:23.782156 14703 ts_tablet_manager.cc:1403] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.005s
I20251211 08:57:23.801941 14429 tablet_service.cc:1505] Processing CreateTablet for tablet e268fd8444ed4974bd6c7b4c5b7c63ce (DEFAULT_TABLE table=default.Table_ed5bf9b8a1c94f46ba4e5184aef6b38d [id=d25467f799644415bf42e502694acce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:23.804169 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e268fd8444ed4974bd6c7b4c5b7c63ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:23.816314 14703 raft_consensus.cc:359] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:23.818513 14703 raft_consensus.cc:385] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:23.819375 14703 raft_consensus.cc:740] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
W20251211 08:57:23.820626 14543 connection.cc:537] client connection to 127.13.146.62:39015 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:57:23.821234 14384 connection.cc:537] client connection to 127.13.146.62:39015 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:57:23.821485 14655 heartbeater.cc:646] Failed to heartbeat to 127.13.146.62:39015 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:57:23.822851 14497 heartbeater.cc:646] Failed to heartbeat to 127.13.146.62:39015 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:57:23.823720 13999 connection.cc:537] client connection to 127.13.146.62:39015 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:57:23.824199 13999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 6 similar messages]
I20251211 08:57:23.819960 14279 tablet_service.cc:1505] Processing CreateTablet for tablet e268fd8444ed4974bd6c7b4c5b7c63ce (DEFAULT_TABLE table=default.Table_ed5bf9b8a1c94f46ba4e5184aef6b38d [id=d25467f799644415bf42e502694acce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:23.827443 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e268fd8444ed4974bd6c7b4c5b7c63ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:23.835434 14703 consensus_queue.cc:260] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:23.846967 14703 ts_tablet_manager.cc:1434] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.064s user 0.030s sys 0.008s
I20251211 08:57:23.851529 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet e268fd8444ed4974bd6c7b4c5b7c63ce (table default.Table_8518029aae6a499fbca00760d422677b [id=d25467f799644415bf42e502694acce4]). Expected version 1 got 0
I20251211 08:57:23.856863 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 19b6f3954b294c49ba0688174aa87f5d (DEFAULT_TABLE table=default.Table_2114d5bcfd134d01b68352cbe97e506f [id=35601cbbebb14fdda09e3c3d5cc47bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:23.857805 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b6f3954b294c49ba0688174aa87f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:23.858330 14278 tablet_service.cc:1505] Processing CreateTablet for tablet 19b6f3954b294c49ba0688174aa87f5d (DEFAULT_TABLE table=default.Table_2114d5bcfd134d01b68352cbe97e506f [id=35601cbbebb14fdda09e3c3d5cc47bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:23.860033 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b6f3954b294c49ba0688174aa87f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:23.866000 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 19b6f3954b294c49ba0688174aa87f5d (DEFAULT_TABLE table=default.Table_2114d5bcfd134d01b68352cbe97e506f [id=35601cbbebb14fdda09e3c3d5cc47bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:23.867066 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b6f3954b294c49ba0688174aa87f5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:23.879074 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.62:39015, user_credentials={real_user=slave}} blocked reactor thread for 54871.2us
I20251211 08:57:23.883616 14714 tablet_bootstrap.cc:492] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:23.884199 14703 tablet_bootstrap.cc:492] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:23.890342 14703 tablet_bootstrap.cc:654] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:23.899482 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_1ceb0e35a939466db7ce05224bf6caaf" } new_table_name: "default.Table_07efa184e6744e9cbbd8e869edb7a3b2" modify_external_catalogs: true
I20251211 08:57:23.902987 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f8b90871ac1a41f6aa1540128ed52976" } modify_external_catalogs: true
I20251211 08:57:23.903671 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f8b90871ac1a41f6aa1540128ed52976" } modify_external_catalogs: true
W20251211 08:57:23.908327 13999 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 -> Peer 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015): Couldn't send request to peer 73702d54989f4959acfc4a6fdcede6ca. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:23.924288 14714 tablet_bootstrap.cc:654] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:23.926303 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e268fd8444ed4974bd6c7b4c5b7c63ce consensus state has no leader
I20251211 08:57:23.934942 14703 tablet_bootstrap.cc:492] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:23.935879 14703 ts_tablet_manager.cc:1403] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.052s user 0.017s sys 0.024s
I20251211 08:57:23.935781 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 19b6f3954b294c49ba0688174aa87f5d
I20251211 08:57:23.938460 14703 raft_consensus.cc:359] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:23.939121 14703 raft_consensus.cc:385] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:23.939046 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet 19b6f3954b294c49ba0688174aa87f5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:23 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:23.939388 14703 raft_consensus.cc:740] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:23.941068 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet 19b6f3954b294c49ba0688174aa87f5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:23 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:23.939936 14703 consensus_queue.cc:260] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:23.951035 14703 ts_tablet_manager.cc:1434] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.015s user 0.002s sys 0.003s
W20251211 08:57:23.954345 14661 tablet.cc:2378] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251211 08:57:23.929553 14714 log.cc:826] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256: Log is configured to *not* fsync() on all Append() calls
W20251211 08:57:23.965056 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet afdfd49bbb9742c69c66d14a35d6f3a0 has no consensus state
I20251211 08:57:23.966972 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 19b6f3954b294c49ba0688174aa87f5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:23 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:23.967520 14717 tablet_replica.cc:333] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:23.968390 14717 raft_consensus.cc:2243] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:23.968917 14717 raft_consensus.cc:2272] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:23.968981 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet e268fd8444ed4974bd6c7b4c5b7c63ce (table default.Table_5232378b2b1448838dc4b77baf176467 [id=d25467f799644415bf42e502694acce4]). Expected version 2 got 0
W20251211 08:57:23.972544 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 19b6f3954b294c49ba0688174aa87f5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b6f3954b294c49ba0688174aa87f5d already in progress: creating tablet
I20251211 08:57:23.973389 14714 tablet_bootstrap.cc:492] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:23.975598 14717 ts_tablet_manager.cc:1916] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:23.976303 14714 ts_tablet_manager.cc:1403] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.093s user 0.017s sys 0.000s
I20251211 08:57:23.991580 14717 ts_tablet_manager.cc:1929] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:23.992126 14717 log.cc:1199] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/19b6f3954b294c49ba0688174aa87f5d
W20251211 08:57:23.992247 14349 heartbeater.cc:646] Failed to heartbeat to 127.13.146.62:39015 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111)
I20251211 08:57:23.993129 14717 ts_tablet_manager.cc:1950] T 19b6f3954b294c49ba0688174aa87f5d P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:23.996523 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 19b6f3954b294c49ba0688174aa87f5d (table default.Table_2114d5bcfd134d01b68352cbe97e506f [id=35601cbbebb14fdda09e3c3d5cc47bd7]) successfully deleted
I20251211 08:57:24.012446 14010 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 258fcf61ef5e448a82bd480d9419c864
W20251211 08:57:24.027963 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 19b6f3954b294c49ba0688174aa87f5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b6f3954b294c49ba0688174aa87f5d already in progress: creating tablet
I20251211 08:57:24.030303 14714 raft_consensus.cc:359] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.032188 14714 raft_consensus.cc:385] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.032624 14714 raft_consensus.cc:740] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.035993 14714 consensus_queue.cc:260] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.036994 14722 tablet_bootstrap.cc:492] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:24.044610 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet 19b6f3954b294c49ba0688174aa87f5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:23 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:24.046752 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 19b6f3954b294c49ba0688174aa87f5d because the tablet was not found. No further retry: Not found: Tablet not found: 19b6f3954b294c49ba0688174aa87f5d
I20251211 08:57:24.048629 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_189349412d134e558d45bbf547051fc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:24.051609 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_189349412d134e558d45bbf547051fc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.049831 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_60804c54b0cb4d838868aa67836a655c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:24.053139 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60804c54b0cb4d838868aa67836a655c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.053556 14714 ts_tablet_manager.cc:1434] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.075s user 0.027s sys 0.013s
I20251211 08:57:24.055009 14714 tablet_bootstrap.cc:492] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:24.061618 14722 tablet_bootstrap.cc:654] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.064524 14722 log.cc:826] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b: Log is configured to *not* fsync() on all Append() calls
I20251211 08:57:24.064930 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet 19b6f3954b294c49ba0688174aa87f5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:23 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:24.065794 14718 tablet_replica.cc:333] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:24.066578 14718 raft_consensus.cc:2243] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:24.067154 14718 raft_consensus.cc:2272] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:24.068445 14279 tablet_service.cc:1505] Processing CreateTablet for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (DEFAULT_TABLE table=default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.069459 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afdfd49bbb9742c69c66d14a35d6f3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.070272 14718 ts_tablet_manager.cc:1916] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:24.076388 14428 tablet_service.cc:1505] Processing CreateTablet for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (DEFAULT_TABLE table=default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.077041 14588 tablet_service.cc:1505] Processing CreateTablet for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (DEFAULT_TABLE table=default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.077327 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afdfd49bbb9742c69c66d14a35d6f3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.078032 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afdfd49bbb9742c69c66d14a35d6f3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.081164 14714 tablet_bootstrap.cc:654] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.099788 14703 tablet_bootstrap.cc:492] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:24.102098 14722 tablet_bootstrap.cc:492] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:24.102681 14722 ts_tablet_manager.cc:1403] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.069s user 0.015s sys 0.005s
I20251211 08:57:24.105688 14703 tablet_bootstrap.cc:654] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.108297 14718 ts_tablet_manager.cc:1929] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:24.108703 14718 log.cc:1199] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/19b6f3954b294c49ba0688174aa87f5d
I20251211 08:57:24.109525 14718 ts_tablet_manager.cc:1950] T 19b6f3954b294c49ba0688174aa87f5d P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:24.112983 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 19b6f3954b294c49ba0688174aa87f5d (table default.Table_2114d5bcfd134d01b68352cbe97e506f [id=35601cbbebb14fdda09e3c3d5cc47bd7]) successfully deleted
I20251211 08:57:24.122444 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ea8ceaf3dcc7478ea8aceec86870f8dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:24.124590 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea8ceaf3dcc7478ea8aceec86870f8dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.142207 14722 raft_consensus.cc:359] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.143355 14722 raft_consensus.cc:385] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.143654 14722 raft_consensus.cc:740] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.143849 14009 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_5232378b2b1448838dc4b77baf176467" } modify_external_catalogs: true
I20251211 08:57:24.144490 14009 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_5232378b2b1448838dc4b77baf176467" } modify_external_catalogs: true
I20251211 08:57:24.144618 14722 consensus_queue.cc:260] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.147804 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b4370c3e38fa4981a2782889fe923f01" } new_table_name: "default.Table_e0ea828152904074a525bb02f73de7c3" modify_external_catalogs: true
I20251211 08:57:24.153483 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_792e39ba352c4fe2844d61c7804e0c42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:24.155182 14722 ts_tablet_manager.cc:1434] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.052s user 0.034s sys 0.007s
W20251211 08:57:24.157372 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_792e39ba352c4fe2844d61c7804e0c42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.163122 14722 tablet_bootstrap.cc:492] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:24.170969 14703 tablet_bootstrap.cc:492] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:24.171418 14703 ts_tablet_manager.cc:1403] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.072s user 0.016s sys 0.019s
I20251211 08:57:24.171706 14722 tablet_bootstrap.cc:654] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.173943 14703 raft_consensus.cc:359] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.174623 14703 raft_consensus.cc:385] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.174914 14703 raft_consensus.cc:740] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.175616 14703 consensus_queue.cc:260] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.176741 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_60804c54b0cb4d838868aa67836a655c" } new_table_name: "default.Table_3124cc6b31254479ae374574e383bccd" modify_external_catalogs: true
I20251211 08:57:24.185648 14010 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]). Expected version 1 got 0
I20251211 08:57:24.187839 14588 tablet_service.cc:1505] Processing CreateTablet for tablet fd873843819447f9b1afc1459188e3fd (DEFAULT_TABLE table=default.Table_b4370c3e38fa4981a2782889fe923f01 [id=f770a7a182474278b0f7533de8640a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.188911 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd873843819447f9b1afc1459188e3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.192425 14428 tablet_service.cc:1505] Processing CreateTablet for tablet fd873843819447f9b1afc1459188e3fd (DEFAULT_TABLE table=default.Table_b4370c3e38fa4981a2782889fe923f01 [id=f770a7a182474278b0f7533de8640a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.194998 14681 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a4c9cd45be3947d09423ed4508ca2e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:24.195641 14681 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 [sys.catalog]: This master's current role is: LEADER
I20251211 08:57:24.196620 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd873843819447f9b1afc1459188e3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.196645 14279 tablet_service.cc:1505] Processing CreateTablet for tablet fd873843819447f9b1afc1459188e3fd (DEFAULT_TABLE table=default.Table_b4370c3e38fa4981a2782889fe923f01 [id=f770a7a182474278b0f7533de8640a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.197580 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd873843819447f9b1afc1459188e3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.200773 14277 tablet_service.cc:1505] Processing CreateTablet for tablet d6c52b08ccaf49c48fe909b72a5da45f (DEFAULT_TABLE table=default.Table_534b28534eba477e8364a8648fba0c60 [id=704c4d83c2464876a16db15b76817d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.201864 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c52b08ccaf49c48fe909b72a5da45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.204622 14429 tablet_service.cc:1505] Processing CreateTablet for tablet d6c52b08ccaf49c48fe909b72a5da45f (DEFAULT_TABLE table=default.Table_534b28534eba477e8364a8648fba0c60 [id=704c4d83c2464876a16db15b76817d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.205729 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c52b08ccaf49c48fe909b72a5da45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.212136 14587 tablet_service.cc:1505] Processing CreateTablet for tablet d6c52b08ccaf49c48fe909b72a5da45f (DEFAULT_TABLE table=default.Table_534b28534eba477e8364a8648fba0c60 [id=704c4d83c2464876a16db15b76817d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.213292 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c52b08ccaf49c48fe909b72a5da45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.226406 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet e268fd8444ed4974bd6c7b4c5b7c63ce
I20251211 08:57:24.229254 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet e268fd8444ed4974bd6c7b4c5b7c63ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:24.229972 14717 tablet_replica.cc:333] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:24.230620 14717 raft_consensus.cc:2243] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:24.231173 14717 raft_consensus.cc:2272] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:24.232486 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet e268fd8444ed4974bd6c7b4c5b7c63ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:24.233387 14717 ts_tablet_manager.cc:1916] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:24.233520 14720 tablet_replica.cc:333] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:24.234316 14720 raft_consensus.cc:2243] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:24.234825 14720 raft_consensus.cc:2272] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:24.237469 14720 ts_tablet_manager.cc:1916] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:24.243768 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_189349412d134e558d45bbf547051fc2" } modify_external_catalogs: true
I20251211 08:57:24.244372 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_189349412d134e558d45bbf547051fc2" } modify_external_catalogs: true
I20251211 08:57:24.248409 14703 ts_tablet_manager.cc:1434] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.077s user 0.007s sys 0.000s
W20251211 08:57:24.248750 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd873843819447f9b1afc1459188e3fd consensus state has no leader
I20251211 08:57:24.249161 14722 tablet_bootstrap.cc:492] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
W20251211 08:57:24.252341 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7eff28d3edb45f7a9733fd6ea1d3d08 has no consensus state
I20251211 08:57:24.258203 14722 ts_tablet_manager.cc:1403] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.095s user 0.014s sys 0.005s
I20251211 08:57:24.261767 14722 raft_consensus.cc:359] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.263182 14703 tablet_bootstrap.cc:492] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:24.264153 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet e268fd8444ed4974bd6c7b4c5b7c63ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:24.265713 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet e268fd8444ed4974bd6c7b4c5b7c63ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e268fd8444ed4974bd6c7b4c5b7c63ce already in progress: creating tablet
I20251211 08:57:24.272099 14714 tablet_bootstrap.cc:492] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:24.272537 14714 ts_tablet_manager.cc:1403] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.218s user 0.007s sys 0.009s
I20251211 08:57:24.274940 14714 raft_consensus.cc:359] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.275514 14714 raft_consensus.cc:385] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.275787 14714 raft_consensus.cc:740] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.279042 14714 consensus_queue.cc:260] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.284997 14714 ts_tablet_manager.cc:1434] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20251211 08:57:24.285791 14714 tablet_bootstrap.cc:492] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:24.288432 14703 tablet_bootstrap.cc:654] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.294971 14717 ts_tablet_manager.cc:1929] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:24.295377 14717 log.cc:1199] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/e268fd8444ed4974bd6c7b4c5b7c63ce
I20251211 08:57:24.296033 14717 ts_tablet_manager.cc:1950] T e268fd8444ed4974bd6c7b4c5b7c63ce P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:24.298492 14010 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet b7877a8696f84515918e69694ae3a6a2
I20251211 08:57:24.298419 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet e268fd8444ed4974bd6c7b4c5b7c63ce (table default.Table_5232378b2b1448838dc4b77baf176467 [id=d25467f799644415bf42e502694acce4]) successfully deleted
I20251211 08:57:24.311058 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_03183c7ccddb49f3b99b80d9133b5319"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:24.312399 14007 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]). Expected version 1 got 0
W20251211 08:57:24.312829 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03183c7ccddb49f3b99b80d9133b5319 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.315281 14588 tablet_service.cc:1505] Processing CreateTablet for tablet d582d092ce05451e9ef541c56de6e387 (DEFAULT_TABLE table=default.Table_ea8ceaf3dcc7478ea8aceec86870f8dd [id=deaf44db4c1146a8ac640aa3ef778467]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.315660 14722 raft_consensus.cc:385] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.316010 14722 raft_consensus.cc:740] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.316363 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d582d092ce05451e9ef541c56de6e387. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.316633 14722 consensus_queue.cc:260] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.319509 14277 tablet_service.cc:1505] Processing CreateTablet for tablet d582d092ce05451e9ef541c56de6e387 (DEFAULT_TABLE table=default.Table_ea8ceaf3dcc7478ea8aceec86870f8dd [id=deaf44db4c1146a8ac640aa3ef778467]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.320533 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d582d092ce05451e9ef541c56de6e387. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.323575 14279 tablet_service.cc:1505] Processing CreateTablet for tablet d7eff28d3edb45f7a9733fd6ea1d3d08 (DEFAULT_TABLE table=default.Table_3124cc6b31254479ae374574e383bccd [id=4be9f85009f34348ac6efa5d5bcb4b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.324519 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7eff28d3edb45f7a9733fd6ea1d3d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.328052 14587 tablet_service.cc:1505] Processing CreateTablet for tablet d7eff28d3edb45f7a9733fd6ea1d3d08 (DEFAULT_TABLE table=default.Table_3124cc6b31254479ae374574e383bccd [id=4be9f85009f34348ac6efa5d5bcb4b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.328603 14429 tablet_service.cc:1505] Processing CreateTablet for tablet d582d092ce05451e9ef541c56de6e387 (DEFAULT_TABLE table=default.Table_ea8ceaf3dcc7478ea8aceec86870f8dd [id=deaf44db4c1146a8ac640aa3ef778467]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.329046 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7eff28d3edb45f7a9733fd6ea1d3d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.329766 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d582d092ce05451e9ef541c56de6e387. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.343603 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_20c0c6a80c5b467a917c0a9e8618440b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:24.345134 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20c0c6a80c5b467a917c0a9e8618440b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.354168 14714 tablet_bootstrap.cc:654] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.362943 14722 ts_tablet_manager.cc:1434] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.104s user 0.005s sys 0.001s
I20251211 08:57:24.369166 14703 tablet_bootstrap.cc:492] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:24.363468 14720 ts_tablet_manager.cc:1929] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:24.370045 14720 log.cc:1199] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/e268fd8444ed4974bd6c7b4c5b7c63ce
I20251211 08:57:24.371337 14720 ts_tablet_manager.cc:1950] T e268fd8444ed4974bd6c7b4c5b7c63ce P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:24.375169 14703 ts_tablet_manager.cc:1403] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.017s
I20251211 08:57:24.377585 14703 raft_consensus.cc:359] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.378229 14703 raft_consensus.cc:385] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.378479 14703 raft_consensus.cc:740] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.379137 14703 consensus_queue.cc:260] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.380463 14722 tablet_bootstrap.cc:492] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:24.388345 14353 tablet.cc:2378] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251211 08:57:24.389799 14703 ts_tablet_manager.cc:1434] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.014s user 0.004s sys 0.003s
I20251211 08:57:24.391424 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet e268fd8444ed4974bd6c7b4c5b7c63ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:24.392263 14718 tablet_replica.cc:333] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:24.392921 14718 raft_consensus.cc:2243] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:24.393466 14718 raft_consensus.cc:2272] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:24.395344 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 19b6f3954b294c49ba0688174aa87f5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:23 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:24.395576 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet e268fd8444ed4974bd6c7b4c5b7c63ce (table default.Table_5232378b2b1448838dc4b77baf176467 [id=d25467f799644415bf42e502694acce4]) successfully deleted
I20251211 08:57:24.395839 14703 tablet_bootstrap.cc:492] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:24.398300 14427 tablet_service.cc:1505] Processing CreateTablet for tablet d7eff28d3edb45f7a9733fd6ea1d3d08 (DEFAULT_TABLE table=default.Table_3124cc6b31254479ae374574e383bccd [id=4be9f85009f34348ac6efa5d5bcb4b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.399520 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7eff28d3edb45f7a9733fd6ea1d3d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.402534 14703 tablet_bootstrap.cc:654] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.408056 14720 tablet_replica.cc:333] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:24.408829 14720 raft_consensus.cc:2243] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:24.409384 14720 raft_consensus.cc:2272] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:24.423879 14720 ts_tablet_manager.cc:1916] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:24.407658 14722 tablet_bootstrap.cc:654] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.441440 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 31efd2ed48e24734af60be837caa8900 (DEFAULT_TABLE table=default.Table_792e39ba352c4fe2844d61c7804e0c42 [id=b207528409f24aa59e806582028f280b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.442512 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31efd2ed48e24734af60be837caa8900. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.449040 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 31efd2ed48e24734af60be837caa8900 (DEFAULT_TABLE table=default.Table_792e39ba352c4fe2844d61c7804e0c42 [id=b207528409f24aa59e806582028f280b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.450033 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31efd2ed48e24734af60be837caa8900. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.452108 14718 ts_tablet_manager.cc:1916] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:24.453704 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_9dad41987e974984812953a74103f5f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:24.455438 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dad41987e974984812953a74103f5f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.458467 14011 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e0ea828152904074a525bb02f73de7c3" } modify_external_catalogs: true
I20251211 08:57:24.458994 14011 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e0ea828152904074a525bb02f73de7c3" } modify_external_catalogs: true
I20251211 08:57:24.462447 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_3124cc6b31254479ae374574e383bccd" } new_table_name: "default.Table_ac054198296e4a67946c02d871076d57" modify_external_catalogs: true
I20251211 08:57:24.465611 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_03183c7ccddb49f3b99b80d9133b5319" } new_table_name: "default.Table_45109de5b01c40d98407cbe7c0de4fe4" modify_external_catalogs: true
I20251211 08:57:24.471683 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4880c750093346c5982eb2d9a7cafa20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:24.473765 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4880c750093346c5982eb2d9a7cafa20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.473834 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 31efd2ed48e24734af60be837caa8900 (DEFAULT_TABLE table=default.Table_792e39ba352c4fe2844d61c7804e0c42 [id=b207528409f24aa59e806582028f280b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.478722 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31efd2ed48e24734af60be837caa8900. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.492542 14712 raft_consensus.cc:493] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:24.493108 14712 raft_consensus.cc:515] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.497895 14712 leader_election.cc:290] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:24.507932 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_792e39ba352c4fe2844d61c7804e0c42" } modify_external_catalogs: true
I20251211 08:57:24.508565 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_792e39ba352c4fe2844d61c7804e0c42" } modify_external_catalogs: true
I20251211 08:57:24.524736 14720 ts_tablet_manager.cc:1929] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:24.529779 14714 tablet_bootstrap.cc:492] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:24.531900 14720 log.cc:1199] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/19b6f3954b294c49ba0688174aa87f5d
I20251211 08:57:24.533289 14720 ts_tablet_manager.cc:1950] T 19b6f3954b294c49ba0688174aa87f5d P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:24.535856 14714 ts_tablet_manager.cc:1403] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.250s user 0.017s sys 0.006s
I20251211 08:57:24.546144 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 19b6f3954b294c49ba0688174aa87f5d (table default.Table_2114d5bcfd134d01b68352cbe97e506f [id=35601cbbebb14fdda09e3c3d5cc47bd7]) successfully deleted
I20251211 08:57:24.553483 14714 raft_consensus.cc:359] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.556397 14714 raft_consensus.cc:385] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.556716 14714 raft_consensus.cc:740] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.557490 14714 consensus_queue.cc:260] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.567193 14714 ts_tablet_manager.cc:1434] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.030s user 0.001s sys 0.005s
I20251211 08:57:24.572355 14714 tablet_bootstrap.cc:492] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:24.568851 14718 ts_tablet_manager.cc:1929] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:24.575011 14718 log.cc:1199] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/e268fd8444ed4974bd6c7b4c5b7c63ce
I20251211 08:57:24.575736 14718 ts_tablet_manager.cc:1950] T e268fd8444ed4974bd6c7b4c5b7c63ce P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:24.578157 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet e268fd8444ed4974bd6c7b4c5b7c63ce (table default.Table_5232378b2b1448838dc4b77baf176467 [id=d25467f799644415bf42e502694acce4]) successfully deleted
I20251211 08:57:24.583827 14723 raft_consensus.cc:493] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:24.583969 14011 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet fd873843819447f9b1afc1459188e3fd
I20251211 08:57:24.584344 14723 raft_consensus.cc:515] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.587283 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet fd873843819447f9b1afc1459188e3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:24.588927 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet fd873843819447f9b1afc1459188e3fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd873843819447f9b1afc1459188e3fd already in progress: creating tablet
I20251211 08:57:24.596325 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet fd873843819447f9b1afc1459188e3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:24.596848 14723 leader_election.cc:290] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:24.599721 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet fd873843819447f9b1afc1459188e3fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd873843819447f9b1afc1459188e3fd already in progress: creating tablet
I20251211 08:57:24.605546 14714 tablet_bootstrap.cc:654] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:24.618758 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet fd873843819447f9b1afc1459188e3fd (table default.Table_e0ea828152904074a525bb02f73de7c3 [id=f770a7a182474278b0f7533de8640a01], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fd873843819447f9b1afc1459188e3fd consensus state has no leader
I20251211 08:57:24.619402 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afdfd49bbb9742c69c66d14a35d6f3a0" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:24.620090 14608 raft_consensus.cc:2468] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:24.621646 14382 leader_election.cc:304] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:24.622756 14723 raft_consensus.cc:2804] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:24.623209 14723 raft_consensus.cc:493] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:24.623651 14723 raft_consensus.cc:3060] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:24.627939 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet fd873843819447f9b1afc1459188e3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:24.630020 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet fd873843819447f9b1afc1459188e3fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd873843819447f9b1afc1459188e3fd already in progress: creating tablet
I20251211 08:57:24.632647 14723 raft_consensus.cc:515] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:24.636323 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7eff28d3edb45f7a9733fd6ea1d3d08 consensus state has no leader
I20251211 08:57:24.636195 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afdfd49bbb9742c69c66d14a35d6f3a0" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:24.636754 14608 raft_consensus.cc:3060] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:24.638723 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afdfd49bbb9742c69c66d14a35d6f3a0" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:24.640568 14385 leader_election.cc:343] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:24.640708 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afdfd49bbb9742c69c66d14a35d6f3a0" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:24.641871 14385 leader_election.cc:343] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:24.637815 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72789234f9e44b82ba438ce478d8372c has no consensus state
I20251211 08:57:24.645431 14608 raft_consensus.cc:2468] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:24.646492 14382 leader_election.cc:304] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:24.648262 14714 tablet_bootstrap.cc:492] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:24.648742 14714 ts_tablet_manager.cc:1403] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.077s user 0.008s sys 0.004s
I20251211 08:57:24.650004 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ead5cbe578404244a3c38f49d8dbc688"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:24.651535 14723 leader_election.cc:290] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:24.652261 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_8e32b532fa954510a2efbc5305ae0aeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:24.651731 14745 raft_consensus.cc:2804] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:24.653628 14745 raft_consensus.cc:697] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:24.660588 14712 raft_consensus.cc:493] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:24.660907 14722 tablet_bootstrap.cc:492] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:24.661299 14722 ts_tablet_manager.cc:1403] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.281s user 0.043s sys 0.006s
I20251211 08:57:24.661000 14712 raft_consensus.cc:515] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.651436 14714 raft_consensus.cc:359] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.662830 14712 leader_election.cc:290] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:24.662968 14714 raft_consensus.cc:385] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.663313 14714 raft_consensus.cc:740] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.663503 14745 consensus_queue.cc:237] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.663965 14714 consensus_queue.cc:260] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.666146 14714 ts_tablet_manager.cc:1434] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.017s user 0.004s sys 0.001s
W20251211 08:57:24.668650 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ead5cbe578404244a3c38f49d8dbc688 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.669657 14714 tablet_bootstrap.cc:492] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:24.671339 14722 raft_consensus.cc:359] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.672304 14722 raft_consensus.cc:385] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251211 08:57:24.669163 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e32b532fa954510a2efbc5305ae0aeb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.673695 14722 raft_consensus.cc:740] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.675086 14722 consensus_queue.cc:260] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.687386 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afdfd49bbb9742c69c66d14a35d6f3a0" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:24.688602 14714 tablet_bootstrap.cc:654] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.689260 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6c52b08ccaf49c48fe909b72a5da45f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:24.689939 14448 raft_consensus.cc:2468] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:24.679131 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afdfd49bbb9742c69c66d14a35d6f3a0" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:24.680234 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6c52b08ccaf49c48fe909b72a5da45f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:24.692078 14542 leader_election.cc:304] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:24.692523 14298 raft_consensus.cc:2468] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:24.692978 14712 raft_consensus.cc:2804] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:24.693307 14712 raft_consensus.cc:493] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:24.693606 14712 raft_consensus.cc:3060] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:24.694317 14722 ts_tablet_manager.cc:1434] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
W20251211 08:57:24.696300 14544 leader_election.cc:343] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:24.698501 14722 tablet_bootstrap.cc:492] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:24.699151 14502 tablet.cc:2378] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251211 08:57:24.706322 14544 leader_election.cc:304] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:24.708676 14712 raft_consensus.cc:515] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.711503 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6c52b08ccaf49c48fe909b72a5da45f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:24.712049 14448 raft_consensus.cc:3060] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:24.720920 14448 raft_consensus.cc:2468] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:24.721571 14712 leader_election.cc:290] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:24.722939 14542 leader_election.cc:304] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:24.731675 14712 raft_consensus.cc:2764] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251211 08:57:24.732272 14712 raft_consensus.cc:2804] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:24.732496 14722 tablet_bootstrap.cc:654] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.732775 14714 tablet_bootstrap.cc:492] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:24.732918 14712 raft_consensus.cc:697] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:24.733219 14714 ts_tablet_manager.cc:1403] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.002s
I20251211 08:57:24.735760 14714 raft_consensus.cc:359] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.736559 14714 raft_consensus.cc:385] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.736882 14714 raft_consensus.cc:740] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.737668 14714 consensus_queue.cc:260] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.740682 14712 consensus_queue.cc:237] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.744535 14714 ts_tablet_manager.cc:1434] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20251211 08:57:24.747799 14714 tablet_bootstrap.cc:492] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:24.752413 14011 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]). Expected version 1 got 0
I20251211 08:57:24.753988 14714 tablet_bootstrap.cc:654] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.755411 14010 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]). Expected version 1 got 0
W20251211 08:57:24.761533 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 55282.9us
I20251211 08:57:24.763520 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6c52b08ccaf49c48fe909b72a5da45f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:24.764925 14722 tablet_bootstrap.cc:492] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:24.765507 14722 ts_tablet_manager.cc:1403] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.003s
I20251211 08:57:24.768927 14722 raft_consensus.cc:359] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:24.770040 14544 leader_election.cc:343] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:24.772111 14722 raft_consensus.cc:385] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.773105 14722 raft_consensus.cc:740] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.775394 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 72789234f9e44b82ba438ce478d8372c (DEFAULT_TABLE table=default.Table_45109de5b01c40d98407cbe7c0de4fe4 [id=c110e62110d344be9fbae792cc0cfe62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.775813 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 72789234f9e44b82ba438ce478d8372c (DEFAULT_TABLE table=default.Table_45109de5b01c40d98407cbe7c0de4fe4 [id=c110e62110d344be9fbae792cc0cfe62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.776408 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72789234f9e44b82ba438ce478d8372c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.776722 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72789234f9e44b82ba438ce478d8372c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.778640 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 72789234f9e44b82ba438ce478d8372c (DEFAULT_TABLE table=default.Table_45109de5b01c40d98407cbe7c0de4fe4 [id=c110e62110d344be9fbae792cc0cfe62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.779681 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72789234f9e44b82ba438ce478d8372c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.774674 14722 consensus_queue.cc:260] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.792316 14722 ts_tablet_manager.cc:1434] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.026s user 0.004s sys 0.004s
I20251211 08:57:24.797272 14722 tablet_bootstrap.cc:492] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:24.808589 14722 tablet_bootstrap.cc:654] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.809703 14714 tablet_bootstrap.cc:492] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:24.809105 14010 catalog_manager.cc:5654] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:24.810225 14714 ts_tablet_manager.cc:1403] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.012s
I20251211 08:57:24.811188 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]). Expected version 1 got 0
I20251211 08:57:24.812898 14714 raft_consensus.cc:359] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.813592 14714 raft_consensus.cc:385] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.813863 14714 raft_consensus.cc:740] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.814528 14714 consensus_queue.cc:260] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.823122 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 31efd2ed48e24734af60be837caa8900
I20251211 08:57:24.824260 14703 tablet_bootstrap.cc:492] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:24.824735 14703 ts_tablet_manager.cc:1403] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.430s user 0.061s sys 0.117s
I20251211 08:57:24.825023 14714 ts_tablet_manager.cc:1434] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
I20251211 08:57:24.827232 14703 raft_consensus.cc:359] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.827950 14703 raft_consensus.cc:385] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.828234 14703 raft_consensus.cc:740] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.828770 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet 31efd2ed48e24734af60be837caa8900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:24.830425 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 31efd2ed48e24734af60be837caa8900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31efd2ed48e24734af60be837caa8900 already in progress: creating tablet
I20251211 08:57:24.831563 14714 tablet_bootstrap.cc:492] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:24.831565 14703 consensus_queue.cc:260] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.835776 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9dad41987e974984812953a74103f5f6" } modify_external_catalogs: true
I20251211 08:57:24.836300 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9dad41987e974984812953a74103f5f6" } modify_external_catalogs: true
I20251211 08:57:24.836830 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 31efd2ed48e24734af60be837caa8900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:24.839632 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 31efd2ed48e24734af60be837caa8900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:24.841660 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 31efd2ed48e24734af60be837caa8900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31efd2ed48e24734af60be837caa8900 already in progress: creating tablet
I20251211 08:57:24.843740 14703 ts_tablet_manager.cc:1434] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.019s user 0.006s sys 0.003s
W20251211 08:57:24.843827 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 31efd2ed48e24734af60be837caa8900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31efd2ed48e24734af60be837caa8900 already in progress: creating tablet
I20251211 08:57:24.846931 14703 tablet_bootstrap.cc:492] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:24.852061 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_534b28534eba477e8364a8648fba0c60" } new_table_name: "default.Table_41f281bb140b455592e96cc27d34e411" modify_external_catalogs: true
I20251211 08:57:24.854615 14714 tablet_bootstrap.cc:654] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.854907 14722 tablet_bootstrap.cc:492] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:24.855289 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ac054198296e4a67946c02d871076d57" } new_table_name: "default.Table_8d6ceb6bcb7c48eeb7a85270f04731b2" modify_external_catalogs: true
I20251211 08:57:24.858856 14703 tablet_bootstrap.cc:654] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.860476 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_adee98d887af4b9bb8d59ea922bd36f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:24.862470 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adee98d887af4b9bb8d59ea922bd36f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.864030 14722 ts_tablet_manager.cc:1403] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.010s
I20251211 08:57:24.851598 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4e4d8cdcfcb349e88961d0d211556cc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:24.867784 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e4d8cdcfcb349e88961d0d211556cc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.873001 14722 raft_consensus.cc:359] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.877331 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 479d7fe06a284234aef420e3842f0eab (DEFAULT_TABLE table=default.Table_20c0c6a80c5b467a917c0a9e8618440b [id=615b65833c3446ca94f57fbcf1659333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.878556 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479d7fe06a284234aef420e3842f0eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.883249 14722 raft_consensus.cc:385] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.885489 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 479d7fe06a284234aef420e3842f0eab (DEFAULT_TABLE table=default.Table_20c0c6a80c5b467a917c0a9e8618440b [id=615b65833c3446ca94f57fbcf1659333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.886483 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479d7fe06a284234aef420e3842f0eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.886873 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 479d7fe06a284234aef420e3842f0eab (DEFAULT_TABLE table=default.Table_20c0c6a80c5b467a917c0a9e8618440b [id=615b65833c3446ca94f57fbcf1659333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:24.887948 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479d7fe06a284234aef420e3842f0eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:24.892310 14714 tablet_bootstrap.cc:492] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:24.892757 14714 ts_tablet_manager.cc:1403] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.061s user 0.017s sys 0.005s
I20251211 08:57:24.894990 14714 raft_consensus.cc:359] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.895660 14714 raft_consensus.cc:385] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.895938 14714 raft_consensus.cc:740] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.896607 14714 consensus_queue.cc:260] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.884230 14722 raft_consensus.cc:740] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.899699 14722 consensus_queue.cc:260] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.909574 14714 ts_tablet_manager.cc:1434] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20251211 08:57:24.903496 14722 ts_tablet_manager.cc:1434] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.038s user 0.006s sys 0.001s
I20251211 08:57:24.913021 14722 tablet_bootstrap.cc:492] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:24.924657 14714 tablet_bootstrap.cc:492] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:24.930281 14714 tablet_bootstrap.cc:654] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.938046 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_1bef0864cfa04e77a665ae61d8cccbe3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:24.940375 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7eff28d3edb45f7a9733fd6ea1d3d08 consensus state has no leader
I20251211 08:57:24.943506 14006 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 99323d65031a49389e4680949705d262
W20251211 08:57:24.944990 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bef0864cfa04e77a665ae61d8cccbe3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:24.951582 14714 tablet_bootstrap.cc:492] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:24.951985 14714 ts_tablet_manager.cc:1403] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.005s
I20251211 08:57:24.954169 14714 raft_consensus.cc:359] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.954829 14714 raft_consensus.cc:385] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.955133 14714 raft_consensus.cc:740] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.957293 14722 tablet_bootstrap.cc:654] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:24.960914 14714 consensus_queue.cc:260] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:24.972509 14703 tablet_bootstrap.cc:492] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:24.973390 14703 ts_tablet_manager.cc:1403] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.127s user 0.026s sys 0.024s
I20251211 08:57:24.972851 14714 ts_tablet_manager.cc:1434] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20251211 08:57:24.975675 14703 raft_consensus.cc:359] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.976380 14703 raft_consensus.cc:385] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.976653 14703 raft_consensus.cc:740] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.977526 14722 tablet_bootstrap.cc:492] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:24.977430 14703 consensus_queue.cc:260] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.977975 14722 ts_tablet_manager.cc:1403] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.005s
I20251211 08:57:24.980329 14722 raft_consensus.cc:359] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.981026 14722 raft_consensus.cc:385] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:24.981563 14714 tablet_bootstrap.cc:492] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:24.982297 14722 raft_consensus.cc:740] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:24.983033 14722 consensus_queue.cc:260] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:24.984169 14703 ts_tablet_manager.cc:1434] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.010s user 0.002s sys 0.002s
I20251211 08:57:24.987579 14703 tablet_bootstrap.cc:492] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:24.989580 14722 ts_tablet_manager.cc:1434] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20251211 08:57:24.990510 14722 tablet_bootstrap.cc:492] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:25.000020 14714 tablet_bootstrap.cc:654] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.000989 14703 tablet_bootstrap.cc:654] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.009083 14722 tablet_bootstrap.cc:654] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.020351 14703 tablet_bootstrap.cc:492] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:25.020805 14703 ts_tablet_manager.cc:1403] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.002s
I20251211 08:57:25.020951 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_8709763a5bb141899fbb13faa5e4a24d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:25.027141 14722 tablet_bootstrap.cc:492] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:25.027702 14722 ts_tablet_manager.cc:1403] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.008s
I20251211 08:57:25.023113 14703 raft_consensus.cc:359] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
W20251211 08:57:25.029183 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8709763a5bb141899fbb13faa5e4a24d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:25.029364 14703 raft_consensus.cc:385] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:25.029671 14703 raft_consensus.cc:740] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:25.030164 14722 raft_consensus.cc:359] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:25.030725 14722 raft_consensus.cc:385] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:25.030364 14703 consensus_queue.cc:260] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:25.031021 14722 raft_consensus.cc:740] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:25.032682 14722 consensus_queue.cc:260] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:25.034399 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]). Expected version 1 got 0
I20251211 08:57:25.037711 14722 ts_tablet_manager.cc:1434] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20251211 08:57:25.040643 14703 ts_tablet_manager.cc:1434] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20251211 08:57:25.046864 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_45109de5b01c40d98407cbe7c0de4fe4" } modify_external_catalogs: true
I20251211 08:57:25.047518 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_45109de5b01c40d98407cbe7c0de4fe4" } modify_external_catalogs: true
I20251211 08:57:25.048197 14722 tablet_bootstrap.cc:492] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:25.049391 14703 tablet_bootstrap.cc:492] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:25.052155 14005 catalog_manager.cc:5654] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:25.053539 14722 tablet_bootstrap.cc:654] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.060272 14703 tablet_bootstrap.cc:654] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.073940 14722 tablet_bootstrap.cc:492] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:25.074383 14722 ts_tablet_manager.cc:1403] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.014s
I20251211 08:57:25.075129 14703 tablet_bootstrap.cc:492] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:25.075542 14703 ts_tablet_manager.cc:1403] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.003s
I20251211 08:57:25.076685 14722 raft_consensus.cc:359] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.077700 14703 raft_consensus.cc:359] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.078377 14703 raft_consensus.cc:385] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:25.078665 14703 raft_consensus.cc:740] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:25.079367 14703 consensus_queue.cc:260] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.082505 14722 raft_consensus.cc:385] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:25.082818 14722 raft_consensus.cc:740] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:25.083427 14722 consensus_queue.cc:260] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.088188 14722 ts_tablet_manager.cc:1434] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.014s user 0.000s sys 0.004s
I20251211 08:57:25.088956 14722 tablet_bootstrap.cc:492] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:25.097692 14703 ts_tablet_manager.cc:1434] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20251211 08:57:25.104406 14722 tablet_bootstrap.cc:654] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.108346 14009 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet d7eff28d3edb45f7a9733fd6ea1d3d08 (table default.Table_8d6ceb6bcb7c48eeb7a85270f04731b2 [id=4be9f85009f34348ac6efa5d5bcb4b0d]). Expected version 3 got 0
I20251211 08:57:25.116437 14703 tablet_bootstrap.cc:492] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:25.120146 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet fd873843819447f9b1afc1459188e3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:25.120941 14718 tablet_replica.cc:333] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:25.121672 14718 raft_consensus.cc:2243] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:25.122133 14718 raft_consensus.cc:2272] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:25.122524 14722 tablet_bootstrap.cc:492] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:25.123010 14722 ts_tablet_manager.cc:1403] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.000s
I20251211 08:57:25.124456 14718 ts_tablet_manager.cc:1916] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:25.125923 14722 raft_consensus.cc:359] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.128671 14703 tablet_bootstrap.cc:654] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.128989 14722 raft_consensus.cc:385] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:25.129448 14722 raft_consensus.cc:740] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:25.130419 14745 raft_consensus.cc:493] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:25.130892 14745 raft_consensus.cc:515] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:25.130369 14722 consensus_queue.cc:260] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.132890 14745 leader_election.cc:290] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:25.135113 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7eff28d3edb45f7a9733fd6ea1d3d08" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:25.135655 14608 raft_consensus.cc:2468] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:25.136384 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet fd873843819447f9b1afc1459188e3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:25.137081 14720 tablet_replica.cc:333] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:25.137775 14720 raft_consensus.cc:2243] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:25.138149 14720 raft_consensus.cc:2272] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:25.139037 14382 leader_election.cc:304] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:25.139081 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7eff28d3edb45f7a9733fd6ea1d3d08" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:25.139568 14298 raft_consensus.cc:2468] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:25.140100 14745 raft_consensus.cc:2804] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:25.140415 14745 raft_consensus.cc:493] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:25.140712 14745 raft_consensus.cc:3060] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.142657 14720 ts_tablet_manager.cc:1916] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:25.147572 14745 raft_consensus.cc:515] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:25.149717 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4e4d8cdcfcb349e88961d0d211556cc0" } new_table_name: "default.Table_b0b4b0150de34800823b57578db1e8bb" modify_external_catalogs: true
I20251211 08:57:25.149892 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8d6ceb6bcb7c48eeb7a85270f04731b2" } modify_external_catalogs: true
I20251211 08:57:25.150346 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8d6ceb6bcb7c48eeb7a85270f04731b2" } modify_external_catalogs: true
I20251211 08:57:25.152032 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7eff28d3edb45f7a9733fd6ea1d3d08" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:25.152582 14608 raft_consensus.cc:3060] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.157226 14714 tablet_bootstrap.cc:492] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:25.157579 14714 ts_tablet_manager.cc:1403] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.176s user 0.011s sys 0.024s
I20251211 08:57:25.158988 14608 raft_consensus.cc:2468] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:25.159893 14382 leader_election.cc:304] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:25.160044 14714 raft_consensus.cc:359] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.160647 14723 raft_consensus.cc:2804] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:25.160878 14714 raft_consensus.cc:385] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:25.161213 14714 raft_consensus.cc:740] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:25.161988 14714 consensus_queue.cc:260] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.170153 14722 ts_tablet_manager.cc:1434] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.047s user 0.005s sys 0.001s
I20251211 08:57:25.172832 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7eff28d3edb45f7a9733fd6ea1d3d08" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:25.176175 14298 raft_consensus.cc:3060] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.183214 14298 raft_consensus.cc:2468] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:25.183508 14769 raft_consensus.cc:493] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:25.183950 14769 raft_consensus.cc:515] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.184784 14745 leader_election.cc:290] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:25.186426 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72789234f9e44b82ba438ce478d8372c" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:25.186991 14298 raft_consensus.cc:2468] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:25.189566 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72789234f9e44b82ba438ce478d8372c" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:25.198562 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 593df63696a04f85b00c301d93748e80 (DEFAULT_TABLE table=default.Table_8e32b532fa954510a2efbc5305ae0aeb [id=ab93470ba8f54070be718c240d49aa13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.199558 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593df63696a04f85b00c301d93748e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.199857 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet d7eff28d3edb45f7a9733fd6ea1d3d08 (table default.Table_8d6ceb6bcb7c48eeb7a85270f04731b2 [id=4be9f85009f34348ac6efa5d5bcb4b0d]). Expected version 3 got 0
I20251211 08:57:25.202764 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 593df63696a04f85b00c301d93748e80 (DEFAULT_TABLE table=default.Table_8e32b532fa954510a2efbc5305ae0aeb [id=ab93470ba8f54070be718c240d49aa13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.203837 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593df63696a04f85b00c301d93748e80. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:25.200759 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f112603316047289459698ec28b54d4 has no consensus state
I20251211 08:57:25.206612 14279 tablet_service.cc:1552] Processing DeleteTablet for tablet 31efd2ed48e24734af60be837caa8900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:25.214546 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 53fc34c41afe44678b8d6dcf79a31f8b (DEFAULT_TABLE table=default.Table_4880c750093346c5982eb2d9a7cafa20 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.215579 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53fc34c41afe44678b8d6dcf79a31f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.215955 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 53fc34c41afe44678b8d6dcf79a31f8b (DEFAULT_TABLE table=default.Table_4880c750093346c5982eb2d9a7cafa20 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.216900 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53fc34c41afe44678b8d6dcf79a31f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.222709 14586 tablet_service.cc:1505] Processing CreateTablet for tablet e5a41643bad84006819fec986657e4ab (DEFAULT_TABLE table=default.Table_ead5cbe578404244a3c38f49d8dbc688 [id=5ecbf67720c544f1bfed70017354edfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.223783 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a41643bad84006819fec986657e4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.223917 14278 tablet_service.cc:1505] Processing CreateTablet for tablet e5a41643bad84006819fec986657e4ab (DEFAULT_TABLE table=default.Table_ead5cbe578404244a3c38f49d8dbc688 [id=5ecbf67720c544f1bfed70017354edfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.224817 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a41643bad84006819fec986657e4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.235028 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_305ab52f2cd144b4b524e8deb6e1264f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:25.236815 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_305ab52f2cd144b4b524e8deb6e1264f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:25.187513 14769 leader_election.cc:290] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:25.248596 14769 raft_consensus.cc:493] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:25.249173 14769 raft_consensus.cc:515] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:25.188694 14385 leader_election.cc:304] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:25.201510 14720 ts_tablet_manager.cc:1929] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:25.251210 14720 log.cc:1199] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/fd873843819447f9b1afc1459188e3fd
W20251211 08:57:25.251344 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 62700.5us
I20251211 08:57:25.251856 14720 ts_tablet_manager.cc:1950] T fd873843819447f9b1afc1459188e3fd P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:25.253324 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 593df63696a04f85b00c301d93748e80 (DEFAULT_TABLE table=default.Table_8e32b532fa954510a2efbc5305ae0aeb [id=ab93470ba8f54070be718c240d49aa13]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:25.253813 14382 leader_election.cc:343] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:25.255663 14429 tablet_service.cc:1505] Processing CreateTablet for tablet 53fc34c41afe44678b8d6dcf79a31f8b (DEFAULT_TABLE table=default.Table_4880c750093346c5982eb2d9a7cafa20 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:25.255993 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 65322.4us
I20251211 08:57:25.257176 14769 leader_election.cc:290] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:25.257651 14769 raft_consensus.cc:2804] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:25.257491 14428 tablet_service.cc:1505] Processing CreateTablet for tablet e5a41643bad84006819fec986657e4ab (DEFAULT_TABLE table=default.Table_ead5cbe578404244a3c38f49d8dbc688 [id=5ecbf67720c544f1bfed70017354edfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.257961 14769 raft_consensus.cc:493] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:25.235726 14718 ts_tablet_manager.cc:1929] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:25.254608 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593df63696a04f85b00c301d93748e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.259016 14718 log.cc:1199] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/fd873843819447f9b1afc1459188e3fd
I20251211 08:57:25.263125 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53fc34c41afe44678b8d6dcf79a31f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.263988 14718 ts_tablet_manager.cc:1950] T fd873843819447f9b1afc1459188e3fd P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:25.264580 14722 tablet_bootstrap.cc:492] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:25.267087 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet fd873843819447f9b1afc1459188e3fd (table default.Table_e0ea828152904074a525bb02f73de7c3 [id=f770a7a182474278b0f7533de8640a01]) successfully deleted
I20251211 08:57:25.268143 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet d7eff28d3edb45f7a9733fd6ea1d3d08
I20251211 08:57:25.259092 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 31efd2ed48e24734af60be837caa8900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:25.271687 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet d7eff28d3edb45f7a9733fd6ea1d3d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:25.266191 14720 tablet_replica.cc:333] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:25.272749 14720 raft_consensus.cc:2243] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:25.273202 14720 raft_consensus.cc:2272] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:25.277688 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a41643bad84006819fec986657e4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.277499 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d582d092ce05451e9ef541c56de6e387" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:25.278090 14298 raft_consensus.cc:2468] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:25.279117 14385 leader_election.cc:304] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:25.279897 14745 raft_consensus.cc:2804] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:25.280203 14745 raft_consensus.cc:493] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:25.287447 14720 ts_tablet_manager.cc:1916] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:25.296404 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet d7eff28d3edb45f7a9733fd6ea1d3d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:25.299204 14749 raft_consensus.cc:493] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:25.299634 14749 raft_consensus.cc:515] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.301398 14749 leader_election.cc:290] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:25.302470 14717 tablet_replica.cc:333] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:25.302750 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_73aada8b1b844ba5a1460ccc54c3761b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:25.303022 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_1fe18f2fd26847cca06517f3a216c641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:25.303440 14717 raft_consensus.cc:2243] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251211 08:57:25.304466 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73aada8b1b844ba5a1460ccc54c3761b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:25.306444 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31efd2ed48e24734af60be837caa8900" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:25.309288 14722 tablet_bootstrap.cc:654] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.309757 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet 31efd2ed48e24734af60be837caa8900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:25.311506 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_797be8cbacae4c4e8f75090433cb74dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:25.312213 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet fd873843819447f9b1afc1459188e3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:24 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:25.304507 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fe18f2fd26847cca06517f3a216c641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:25.316072 14544 leader_election.cc:343] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: Tablet not RUNNING: STOPPED
I20251211 08:57:25.311668 14717 raft_consensus.cc:2272] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:25.318725 14298 raft_consensus.cc:3060] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.325052 14717 ts_tablet_manager.cc:1916] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:25.326900 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet d7eff28d3edb45f7a9733fd6ea1d3d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:25.313510 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_797be8cbacae4c4e8f75090433cb74dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:25.341162 14298 raft_consensus.cc:1275] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Refusing update from remote peer c5e624bcfd154cd586ef1dedb9940063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:25.342641 14712 consensus_queue.cc:1048] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:25.342826 14010 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_41f281bb140b455592e96cc27d34e411" } new_table_name: "default.Table_bc568f3abe9d41fc84ca9e7c0bf3b28d" modify_external_catalogs: true
I20251211 08:57:25.360507 14703 tablet_bootstrap.cc:492] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:25.361092 14703 ts_tablet_manager.cc:1403] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.245s user 0.004s sys 0.020s
I20251211 08:57:25.363946 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d582d092ce05451e9ef541c56de6e387" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:25.364498 14608 raft_consensus.cc:2468] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:25.274852 14769 raft_consensus.cc:3060] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.283377 14745 raft_consensus.cc:3060] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.370934 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet d7eff28d3edb45f7a9733fd6ea1d3d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:25.375545 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31efd2ed48e24734af60be837caa8900" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:25.376116 14448 raft_consensus.cc:2468] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:25.377563 14449 raft_consensus.cc:1275] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Refusing update from remote peer c5e624bcfd154cd586ef1dedb9940063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:25.381826 14542 leader_election.cc:304] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:25.387323 14748 raft_consensus.cc:2804] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:25.387646 14748 raft_consensus.cc:493] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:25.387935 14748 raft_consensus.cc:3060] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.393689 14297 raft_consensus.cc:3060] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.396211 14748 raft_consensus.cc:515] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.399923 14718 tablet_replica.cc:333] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:25.400673 14718 raft_consensus.cc:2243] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:25.401086 14718 raft_consensus.cc:2272] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:25.401454 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31efd2ed48e24734af60be837caa8900" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:25.401948 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet fd873843819447f9b1afc1459188e3fd (table default.Table_e0ea828152904074a525bb02f73de7c3 [id=f770a7a182474278b0f7533de8640a01]) successfully deleted
W20251211 08:57:25.403146 14542 leader_election.cc:343] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:25.404942 14608 raft_consensus.cc:1275] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Refusing update from remote peer b276f77f87254a5d909c5ab940a39256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:25.405944 14748 leader_election.cc:290] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:25.406364 14748 consensus_queue.cc:1048] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:25.407625 14767 consensus_queue.cc:1048] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:25.411686 14774 tablet.cc:1722] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:25.417321 14703 raft_consensus.cc:359] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.417909 14703 raft_consensus.cc:385] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:25.418129 14703 raft_consensus.cc:740] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:25.426452 14717 ts_tablet_manager.cc:1929] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:25.432137 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31efd2ed48e24734af60be837caa8900" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:25.433840 14544 leader_election.cc:343] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: Tablet not RUNNING: STOPPED
I20251211 08:57:25.434327 14544 leader_election.cc:304] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:25.435349 14712 raft_consensus.cc:2749] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251211 08:57:25.432509 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4880c750093346c5982eb2d9a7cafa20" } new_table_name: "default.Table_877382ed2e4f49a6adc0ef3bf98e9e37" modify_external_catalogs: true
I20251211 08:57:25.439460 14297 raft_consensus.cc:1275] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Refusing update from remote peer b276f77f87254a5d909c5ab940a39256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:25.444386 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]). Expected version 1 got 0
I20251211 08:57:25.446530 14767 consensus_queue.cc:1048] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:25.451367 14722 tablet_bootstrap.cc:492] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:25.451753 14722 ts_tablet_manager.cc:1403] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.193s user 0.020s sys 0.005s
I20251211 08:57:25.456264 14718 ts_tablet_manager.cc:1916] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:25.428354 14703 consensus_queue.cc:260] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.462972 14723 raft_consensus.cc:697] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:25.464242 14703 ts_tablet_manager.cc:1434] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.103s user 0.007s sys 0.000s
I20251211 08:57:25.463862 14723 consensus_queue.cc:237] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:25.469537 14779 tablet.cc:1722] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:25.474191 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 72789234f9e44b82ba438ce478d8372c
I20251211 08:57:25.475380 14703 tablet_bootstrap.cc:492] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:25.475863 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet 72789234f9e44b82ba438ce478d8372c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:25.386394 14769 raft_consensus.cc:515] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.476672 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet 72789234f9e44b82ba438ce478d8372c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:25.479614 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72789234f9e44b82ba438ce478d8372c" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:25.480114 14296 raft_consensus.cc:3060] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.480479 14279 tablet_service.cc:1552] Processing DeleteTablet for tablet 72789234f9e44b82ba438ce478d8372c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:25.484004 14279 tablet_service.cc:1505] Processing CreateTablet for tablet bcdb31a588b84616844986e6101bb154 (DEFAULT_TABLE table=default.Table_8709763a5bb141899fbb13faa5e4a24d [id=5a9a6d684a1a496595f5bb9ddfca3f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.484007 14424 tablet_service.cc:1505] Processing CreateTablet for tablet bcdb31a588b84616844986e6101bb154 (DEFAULT_TABLE table=default.Table_8709763a5bb141899fbb13faa5e4a24d [id=5a9a6d684a1a496595f5bb9ddfca3f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.484930 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcdb31a588b84616844986e6101bb154. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.484975 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcdb31a588b84616844986e6101bb154. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.485985 14588 tablet_service.cc:1505] Processing CreateTablet for tablet bcdb31a588b84616844986e6101bb154 (DEFAULT_TABLE table=default.Table_8709763a5bb141899fbb13faa5e4a24d [id=5a9a6d684a1a496595f5bb9ddfca3f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.486953 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcdb31a588b84616844986e6101bb154. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.487391 14278 tablet_service.cc:1505] Processing CreateTablet for tablet 8f112603316047289459698ec28b54d4 (DEFAULT_TABLE table=default.Table_b0b4b0150de34800823b57578db1e8bb [id=202989bd9772430baafe83f28d3debe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.488276 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f112603316047289459698ec28b54d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.490957 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]). Expected version 1 got 0
I20251211 08:57:25.492758 14296 raft_consensus.cc:2468] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:25.493665 14385 leader_election.cc:304] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:25.494061 14012 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8709763a5bb141899fbb13faa5e4a24d" } modify_external_catalogs: true
I20251211 08:57:25.494540 14767 raft_consensus.cc:2804] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:25.494573 14012 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8709763a5bb141899fbb13faa5e4a24d" } modify_external_catalogs: true
I20251211 08:57:25.495069 14767 raft_consensus.cc:697] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:25.495879 14767 consensus_queue.cc:237] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.504168 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 8f112603316047289459698ec28b54d4 (DEFAULT_TABLE table=default.Table_b0b4b0150de34800823b57578db1e8bb [id=202989bd9772430baafe83f28d3debe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.504647 14777 mvcc.cc:204] Tried to move back new op lower bound from 7231256352310480896 to 7231256349387026432. Current Snapshot: MvccSnapshot[applied={T|T < 7231256352310480896}]
I20251211 08:57:25.505195 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f112603316047289459698ec28b54d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.510875 14717 log.cc:1199] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/d7eff28d3edb45f7a9733fd6ea1d3d08
I20251211 08:57:25.511603 14717 ts_tablet_manager.cc:1950] T d7eff28d3edb45f7a9733fd6ea1d3d08 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:25.513501 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 8f112603316047289459698ec28b54d4 (DEFAULT_TABLE table=default.Table_b0b4b0150de34800823b57578db1e8bb [id=202989bd9772430baafe83f28d3debe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.514096 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet d7eff28d3edb45f7a9733fd6ea1d3d08 (table default.Table_8d6ceb6bcb7c48eeb7a85270f04731b2 [id=4be9f85009f34348ac6efa5d5bcb4b0d]) successfully deleted
I20251211 08:57:25.514497 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f112603316047289459698ec28b54d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.514889 14717 tablet_replica.cc:333] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:25.515636 14717 raft_consensus.cc:2243] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:25.516088 14717 raft_consensus.cc:2272] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:25.520040 14769 leader_election.cc:290] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:25.521262 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_cbeef8d5f95540b9a51b10a0a18a2dce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:25.525696 14010 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet d6c52b08ccaf49c48fe909b72a5da45f (table default.Table_bc568f3abe9d41fc84ca9e7c0bf3b28d [id=704c4d83c2464876a16db15b76817d50]). Expected version 2 got 0
I20251211 08:57:25.526328 14010 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]). Expected version 1 got 0
W20251211 08:57:25.526978 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbeef8d5f95540b9a51b10a0a18a2dce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:25.536070 14720 ts_tablet_manager.cc:1929] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:25.536446 14720 log.cc:1199] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/31efd2ed48e24734af60be837caa8900
I20251211 08:57:25.537123 14720 ts_tablet_manager.cc:1950] T 31efd2ed48e24734af60be837caa8900 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:25.546550 14714 ts_tablet_manager.cc:1434] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.389s user 0.011s sys 0.042s
I20251211 08:57:25.550268 14748 raft_consensus.cc:493] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:25.550691 14748 raft_consensus.cc:515] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:25.552338 14013 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53fc34c41afe44678b8d6dcf79a31f8b consensus state has no leader
I20251211 08:57:25.552661 14748 leader_election.cc:290] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:25.554876 14542 leader_election.cc:343] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: fd873843819447f9b1afc1459188e3fd
W20251211 08:57:25.557031 14544 leader_election.cc:343] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Not found: Tablet not found: fd873843819447f9b1afc1459188e3fd
I20251211 08:57:25.557417 14544 leader_election.cc:304] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:25.558295 14748 raft_consensus.cc:2749] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:25.559671 14703 tablet_bootstrap.cc:654] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.559876 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72789234f9e44b82ba438ce478d8372c" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:25.560425 14607 raft_consensus.cc:3060] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.561335 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_8d88398a2b724366b2350c38c52814f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:25.563266 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d88398a2b724366b2350c38c52814f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:25.566720 14607 raft_consensus.cc:2468] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:25.567888 14720 tablet_replica.cc:333] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:25.568733 14720 raft_consensus.cc:2243] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:25.569237 14720 raft_consensus.cc:2272] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:25.570243 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 31efd2ed48e24734af60be837caa8900 (table default.Table_792e39ba352c4fe2844d61c7804e0c42 [id=b207528409f24aa59e806582028f280b]) successfully deleted
I20251211 08:57:25.574088 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 292b641df22f478d8849ff6148ecf951 (DEFAULT_TABLE table=default.Table_adee98d887af4b9bb8d59ea922bd36f9 [id=1d8b25ddadc64095ad0422142f43812b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.575269 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292b641df22f478d8849ff6148ecf951. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.575248 14717 ts_tablet_manager.cc:1916] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:25.473878 14745 raft_consensus.cc:515] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:25.588588 14780 tablet.cc:1722] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:25.592343 14714 tablet_bootstrap.cc:492] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:25.594026 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 292b641df22f478d8849ff6148ecf951 (DEFAULT_TABLE table=default.Table_adee98d887af4b9bb8d59ea922bd36f9 [id=1d8b25ddadc64095ad0422142f43812b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.595237 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 1a2dcb9ac62e4aac885d638f4e2b072e (DEFAULT_TABLE table=default.Table_1bef0864cfa04e77a665ae61d8cccbe3 [id=1673af33c2f846c7bf092303eff6e8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.596975 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a2dcb9ac62e4aac885d638f4e2b072e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.600844 14745 leader_election.cc:290] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:25.602319 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7ff0193278184b638a2f8e40fd323397"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:25.604228 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ff0193278184b638a2f8e40fd323397 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:25.608242 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292b641df22f478d8849ff6148ecf951. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.608760 14703 tablet_bootstrap.cc:492] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:25.611131 14703 ts_tablet_manager.cc:1403] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.136s user 0.007s sys 0.003s
W20251211 08:57:25.597036 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 89998.3us
I20251211 08:57:25.613379 14703 raft_consensus.cc:359] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.613970 14703 raft_consensus.cc:385] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:25.614217 14703 raft_consensus.cc:740] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:25.614759 14703 consensus_queue.cc:260] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.630743 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 1a2dcb9ac62e4aac885d638f4e2b072e (DEFAULT_TABLE table=default.Table_1bef0864cfa04e77a665ae61d8cccbe3 [id=1673af33c2f846c7bf092303eff6e8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.631912 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a2dcb9ac62e4aac885d638f4e2b072e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.644375 14703 ts_tablet_manager.cc:1434] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.033s user 0.005s sys 0.001s
I20251211 08:57:25.644582 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d582d092ce05451e9ef541c56de6e387" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:25.645077 14607 raft_consensus.cc:3060] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.597842 14722 raft_consensus.cc:359] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.647261 14722 raft_consensus.cc:385] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:25.647526 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d582d092ce05451e9ef541c56de6e387" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:25.648068 14297 raft_consensus.cc:3060] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:25.649646 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet d6c52b08ccaf49c48fe909b72a5da45f (table default.Table_bc568f3abe9d41fc84ca9e7c0bf3b28d [id=704c4d83c2464876a16db15b76817d50]). Expected version 2 got 1
I20251211 08:57:25.654722 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 1a2dcb9ac62e4aac885d638f4e2b072e (DEFAULT_TABLE table=default.Table_1bef0864cfa04e77a665ae61d8cccbe3 [id=1673af33c2f846c7bf092303eff6e8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.655194 14703 tablet_bootstrap.cc:492] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:25.655799 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a2dcb9ac62e4aac885d638f4e2b072e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.647553 14722 raft_consensus.cc:740] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:25.654978 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_877382ed2e4f49a6adc0ef3bf98e9e37" } new_table_name: "default.Table_38d82f258eab426daa74a3a31f588767" modify_external_catalogs: true
I20251211 08:57:25.655952 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6c1b92755bba45abbc6750537eabadb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:25.664987 13997 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 53fc34c41afe44678b8d6dcf79a31f8b (table default.Table_877382ed2e4f49a6adc0ef3bf98e9e37 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 53fc34c41afe44678b8d6dcf79a31f8b consensus state has no leader
W20251211 08:57:25.672142 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c1b92755bba45abbc6750537eabadb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:25.673914 14788 tablet.cc:1722] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:25.657313 14722 consensus_queue.cc:260] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.683954 14771 mvcc.cc:204] Tried to move back new op lower bound from 7231256351434772480 to 7231256349667520512. Current Snapshot: MvccSnapshot[applied={T|T < 7231256351434772480}]
I20251211 08:57:25.691968 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 292b641df22f478d8849ff6148ecf951 (DEFAULT_TABLE table=default.Table_adee98d887af4b9bb8d59ea922bd36f9 [id=1d8b25ddadc64095ad0422142f43812b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:25.694264 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292b641df22f478d8849ff6148ecf951. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:25.717671 14714 tablet_bootstrap.cc:654] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.719487 14774 tablet.cc:1722] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:25.686374 14297 raft_consensus.cc:2468] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:25.723721 14718 ts_tablet_manager.cc:1929] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:25.725013 14722 ts_tablet_manager.cc:1434] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.273s user 0.000s sys 0.013s
I20251211 08:57:25.724344 14385 leader_election.cc:304] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:25.725792 14767 raft_consensus.cc:2804] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:25.726006 14722 tablet_bootstrap.cc:492] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:25.726238 14767 raft_consensus.cc:697] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:25.727908 14767 consensus_queue.cc:237] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:25.729985 14789 tablet.cc:1722] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:25.722287 14720 ts_tablet_manager.cc:1916] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:25.753587 14791 tablet.cc:1722] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:25.725118 14718 log.cc:1199] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/31efd2ed48e24734af60be837caa8900
I20251211 08:57:25.757592 14718 ts_tablet_manager.cc:1950] T 31efd2ed48e24734af60be837caa8900 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:25.769172 14011 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet d6c52b08ccaf49c48fe909b72a5da45f (table default.Table_bc568f3abe9d41fc84ca9e7c0bf3b28d [id=704c4d83c2464876a16db15b76817d50]). Expected version 2 got 0
I20251211 08:57:25.771297 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet d6c52b08ccaf49c48fe909b72a5da45f (table default.Table_bc568f3abe9d41fc84ca9e7c0bf3b28d [id=704c4d83c2464876a16db15b76817d50]). Expected version 2 got 0
I20251211 08:57:25.780635 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_07efa184e6744e9cbbd8e869edb7a3b2" } modify_external_catalogs: true
I20251211 08:57:25.781174 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_07efa184e6744e9cbbd8e869edb7a3b2" } modify_external_catalogs: true
I20251211 08:57:25.785732 14703 tablet_bootstrap.cc:654] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:25.786892 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.62:39015, user_credentials={real_user=slave}} blocked reactor thread for 109503us
I20251211 08:57:25.800799 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 31efd2ed48e24734af60be837caa8900 (table default.Table_792e39ba352c4fe2844d61c7804e0c42 [id=b207528409f24aa59e806582028f280b]) successfully deleted
I20251211 08:57:25.800870 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_73aada8b1b844ba5a1460ccc54c3761b" } new_table_name: "default.Table_eb1d2cda272c4981a164a7db6301ec2c" modify_external_catalogs: true
I20251211 08:57:25.808917 14717 ts_tablet_manager.cc:1929] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:25.809283 14717 log.cc:1199] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/31efd2ed48e24734af60be837caa8900
I20251211 08:57:25.809845 14717 ts_tablet_manager.cc:1950] T 31efd2ed48e24734af60be837caa8900 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:25.813083 14717 tablet_replica.cc:333] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:25.813200 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 31efd2ed48e24734af60be837caa8900 (table default.Table_792e39ba352c4fe2844d61c7804e0c42 [id=b207528409f24aa59e806582028f280b]) successfully deleted
I20251211 08:57:25.813843 14607 raft_consensus.cc:2468] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:25.818099 14718 tablet_replica.cc:333] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:25.813845 14717 raft_consensus.cc:2243] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:25.819363 14717 raft_consensus.cc:2272] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:25.820603 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet bcdb31a588b84616844986e6101bb154
I20251211 08:57:25.821168 14717 ts_tablet_manager.cc:1916] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:25.822455 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet bcdb31a588b84616844986e6101bb154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:25.823107 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet bcdb31a588b84616844986e6101bb154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:25.824052 14279 tablet_service.cc:1552] Processing DeleteTablet for tablet bcdb31a588b84616844986e6101bb154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:25.828647 14718 raft_consensus.cc:2243] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:25.830199 14718 pending_rounds.cc:70] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:25.831327 14718 pending_rounds.cc:77] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256352627548160 op_type: NO_OP noop_request { }
I20251211 08:57:25.830525 14788 tablet.cc:1722] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:25.838953 14718 raft_consensus.cc:2889] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:25.855707 14722 tablet_bootstrap.cc:654] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.859257 14703 tablet_bootstrap.cc:492] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:25.859622 14703 ts_tablet_manager.cc:1403] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.205s user 0.012s sys 0.017s
I20251211 08:57:25.861447 14703 raft_consensus.cc:359] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:25.861964 14703 raft_consensus.cc:385] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:25.862170 14703 raft_consensus.cc:740] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:25.862659 14703 consensus_queue.cc:260] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:25.874923 14703 ts_tablet_manager.cc:1434] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20251211 08:57:25.906644 14717 ts_tablet_manager.cc:1929] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:25.910984 14717 log.cc:1199] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/fd873843819447f9b1afc1459188e3fd
I20251211 08:57:25.911811 14717 ts_tablet_manager.cc:1950] T fd873843819447f9b1afc1459188e3fd P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:25.914513 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet fd873843819447f9b1afc1459188e3fd (table default.Table_e0ea828152904074a525bb02f73de7c3 [id=f770a7a182474278b0f7533de8640a01]) successfully deleted
I20251211 08:57:25.859761 14789 tablet.cc:1722] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:25.927781 14720 ts_tablet_manager.cc:1929] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:25.928140 14720 log.cc:1199] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/d7eff28d3edb45f7a9733fd6ea1d3d08
I20251211 08:57:25.929705 14717 tablet_replica.cc:333] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:57:25.930603 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53fc34c41afe44678b8d6dcf79a31f8b consensus state has no leader
I20251211 08:57:25.931381 14717 raft_consensus.cc:2243] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:25.932158 14720 ts_tablet_manager.cc:1950] T d7eff28d3edb45f7a9733fd6ea1d3d08 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:25.853142 14714 tablet_bootstrap.cc:492] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:25.943111 14714 ts_tablet_manager.cc:1403] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.351s user 0.012s sys 0.005s
I20251211 08:57:25.945185 14714 raft_consensus.cc:359] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.945719 14714 raft_consensus.cc:385] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:25.945928 14714 raft_consensus.cc:740] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:25.946666 14714 consensus_queue.cc:260] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:25.947633 14717 raft_consensus.cc:2272] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:25.948482 14714 ts_tablet_manager.cc:1434] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251211 08:57:25.949250 14714 tablet_bootstrap.cc:492] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:25.949421 14717 ts_tablet_manager.cc:1916] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:25.958603 14714 tablet_bootstrap.cc:654] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.963404 14718 raft_consensus.cc:2272] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:25.973614 14720 tablet_replica.cc:333] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:25.973806 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet d7eff28d3edb45f7a9733fd6ea1d3d08 (table default.Table_8d6ceb6bcb7c48eeb7a85270f04731b2 [id=4be9f85009f34348ac6efa5d5bcb4b0d]) successfully deleted
I20251211 08:57:25.976433 14703 tablet_bootstrap.cc:492] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:25.981555 14703 tablet_bootstrap.cc:654] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:25.986574 14718 ts_tablet_manager.cc:1916] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:25.994608 14703 tablet_bootstrap.cc:492] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:25.995048 14703 ts_tablet_manager.cc:1403] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.005s
I20251211 08:57:25.997282 14717 ts_tablet_manager.cc:1929] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:25.997639 14717 log.cc:1199] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/72789234f9e44b82ba438ce478d8372c
I20251211 08:57:25.998322 14717 ts_tablet_manager.cc:1950] T 72789234f9e44b82ba438ce478d8372c P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:25.973151 14722 tablet_bootstrap.cc:492] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:25.999142 14722 ts_tablet_manager.cc:1403] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.273s user 0.007s sys 0.004s
I20251211 08:57:25.999284 14720 raft_consensus.cc:2243] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:25.999747 14720 raft_consensus.cc:2272] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:26.001215 14720 ts_tablet_manager.cc:1916] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:26.001044 14722 raft_consensus.cc:359] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.001560 14722 raft_consensus.cc:385] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.001756 14722 raft_consensus.cc:740] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.003247 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6169dbdabeee4eaeab82230228af27b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:26.004861 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6169dbdabeee4eaeab82230228af27b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:26.005931 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 72789234f9e44b82ba438ce478d8372c (table default.Table_45109de5b01c40d98407cbe7c0de4fe4 [id=c110e62110d344be9fbae792cc0cfe62]) successfully deleted
W20251211 08:57:26.007387 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet bcdb31a588b84616844986e6101bb154 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcdb31a588b84616844986e6101bb154 already in progress: creating tablet
I20251211 08:57:26.012080 14011 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9] alter complete (version 1)
I20251211 08:57:26.016278 14722 consensus_queue.cc:260] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.018268 14004 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_bc568f3abe9d41fc84ca9e7c0bf3b28d [id=704c4d83c2464876a16db15b76817d50] alter complete (version 2)
I20251211 08:57:26.025452 14703 raft_consensus.cc:359] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
W20251211 08:57:26.031797 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1853070f81c34b52b8c74a7402e6dad3 has no consensus state
I20251211 08:57:26.034602 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8d88398a2b724366b2350c38c52814f3" } modify_external_catalogs: true
I20251211 08:57:26.035066 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8d88398a2b724366b2350c38c52814f3" } modify_external_catalogs: true
I20251211 08:57:26.036249 14703 raft_consensus.cc:385] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.036484 14703 raft_consensus.cc:740] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.037257 14767 raft_consensus.cc:493] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:26.037667 14767 raft_consensus.cc:515] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.039902 14767 leader_election.cc:290] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:26.043133 14276 tablet_service.cc:1505] Processing CreateTablet for tablet c181adf9db574ab39432b2f9b14fe4d7 (DEFAULT_TABLE table=default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.044231 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c181adf9db574ab39432b2f9b14fe4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.043685 14703 consensus_queue.cc:260] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.045917 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "479d7fe06a284234aef420e3842f0eab" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:26.047847 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "479d7fe06a284234aef420e3842f0eab" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:26.048408 14607 raft_consensus.cc:2468] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:26.046480 14299 raft_consensus.cc:2468] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:26.053529 14703 ts_tablet_manager.cc:1434] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.056s user 0.000s sys 0.005s
I20251211 08:57:26.054869 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 7d4f6727c4464cc9b7d68866c3969a0f (DEFAULT_TABLE table=default.Table_1fe18f2fd26847cca06517f3a216c641 [id=e6bc81a67ba949c69798c5049da505b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.055424 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 7d4f6727c4464cc9b7d68866c3969a0f (DEFAULT_TABLE table=default.Table_1fe18f2fd26847cca06517f3a216c641 [id=e6bc81a67ba949c69798c5049da505b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.055866 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4f6727c4464cc9b7d68866c3969a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.056265 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4f6727c4464cc9b7d68866c3969a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.058511 14703 tablet_bootstrap.cc:492] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:26.062749 14279 tablet_service.cc:1505] Processing CreateTablet for tablet ad903727f79a4180adf047c92d981c7e (DEFAULT_TABLE table=default.Table_305ab52f2cd144b4b524e8deb6e1264f [id=2c2c410df07d4e4cbc60e08dce22efe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.063763 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad903727f79a4180adf047c92d981c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.065629 14584 tablet_service.cc:1505] Processing CreateTablet for tablet ad903727f79a4180adf047c92d981c7e (DEFAULT_TABLE table=default.Table_305ab52f2cd144b4b524e8deb6e1264f [id=2c2c410df07d4e4cbc60e08dce22efe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.066526 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad903727f79a4180adf047c92d981c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.071802 14773 raft_consensus.cc:493] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:26.072264 14722 ts_tablet_manager.cc:1434] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.073s user 0.003s sys 0.001s
I20251211 08:57:26.072186 14773 raft_consensus.cc:515] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.073391 14722 tablet_bootstrap.cc:492] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:26.074122 14773 leader_election.cc:290] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:26.075157 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53fc34c41afe44678b8d6dcf79a31f8b" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:26.078805 14544 leader_election.cc:343] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:26.083438 14703 tablet_bootstrap.cc:654] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.084578 14587 tablet_service.cc:1505] Processing CreateTablet for tablet c181adf9db574ab39432b2f9b14fe4d7 (DEFAULT_TABLE table=default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.085590 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c181adf9db574ab39432b2f9b14fe4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.099018 14703 tablet_bootstrap.cc:492] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:26.099372 14703 ts_tablet_manager.cc:1403] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.004s
I20251211 08:57:26.101090 14278 tablet_service.cc:1505] Processing CreateTablet for tablet 7b60d99c2b4f442da4a4d9c5190f5797 (DEFAULT_TABLE table=default.Table_cbeef8d5f95540b9a51b10a0a18a2dce [id=71569e18b5d347598f45f7e38c073515]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.101975 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b60d99c2b4f442da4a4d9c5190f5797. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.104023 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 7b60d99c2b4f442da4a4d9c5190f5797 (DEFAULT_TABLE table=default.Table_cbeef8d5f95540b9a51b10a0a18a2dce [id=71569e18b5d347598f45f7e38c073515]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.105023 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b60d99c2b4f442da4a4d9c5190f5797. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.109764 14703 raft_consensus.cc:359] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.110474 14703 raft_consensus.cc:385] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.113616 14773 raft_consensus.cc:493] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:26.115079 14773 raft_consensus.cc:515] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.117314 14773 leader_election.cc:290] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:26.117983 14703 raft_consensus.cc:740] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.118219 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5a41643bad84006819fec986657e4ab" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:26.118894 14703 consensus_queue.cc:260] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.123718 14703 ts_tablet_manager.cc:1434] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20251211 08:57:26.124435 14703 tablet_bootstrap.cc:492] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:26.127750 14722 tablet_bootstrap.cc:654] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.129146 14703 tablet_bootstrap.cc:654] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.136785 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 1853070f81c34b52b8c74a7402e6dad3 (DEFAULT_TABLE table=default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.045149 14425 tablet_service.cc:1505] Processing CreateTablet for tablet c181adf9db574ab39432b2f9b14fe4d7 (DEFAULT_TABLE table=default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.137677 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1853070f81c34b52b8c74a7402e6dad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.138303 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c181adf9db574ab39432b2f9b14fe4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.164314 14703 tablet_bootstrap.cc:492] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:26.164938 14703 ts_tablet_manager.cc:1403] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.004s
W20251211 08:57:26.166462 13999 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 -> Peer 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015): Couldn't send request to peer 73702d54989f4959acfc4a6fdcede6ca. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251211 08:57:26.167979 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53fc34c41afe44678b8d6dcf79a31f8b" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:26.170552 14542 leader_election.cc:343] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:26.171205 14542 leader_election.cc:304] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:26.172410 14773 raft_consensus.cc:2749] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:26.175622 14703 raft_consensus.cc:359] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.149672 14385 leader_election.cc:304] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:26.176154 14703 raft_consensus.cc:385] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.176390 14703 raft_consensus.cc:740] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.176887 14703 consensus_queue.cc:260] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.178508 14703 ts_tablet_manager.cc:1434] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20251211 08:57:26.179594 14424 tablet_service.cc:1505] Processing CreateTablet for tablet ad903727f79a4180adf047c92d981c7e (DEFAULT_TABLE table=default.Table_305ab52f2cd144b4b524e8deb6e1264f [id=2c2c410df07d4e4cbc60e08dce22efe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.180673 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad903727f79a4180adf047c92d981c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.183512 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 7b60d99c2b4f442da4a4d9c5190f5797 (DEFAULT_TABLE table=default.Table_cbeef8d5f95540b9a51b10a0a18a2dce [id=71569e18b5d347598f45f7e38c073515]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.184662 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b60d99c2b4f442da4a4d9c5190f5797. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.185732 14703 tablet_bootstrap.cc:492] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:26.188071 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 1853070f81c34b52b8c74a7402e6dad3 (DEFAULT_TABLE table=default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.189340 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1853070f81c34b52b8c74a7402e6dad3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:26.191908 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 44607.8us
I20251211 08:57:26.192761 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 7d4f6727c4464cc9b7d68866c3969a0f (DEFAULT_TABLE table=default.Table_1fe18f2fd26847cca06517f3a216c641 [id=e6bc81a67ba949c69798c5049da505b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.193974 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4f6727c4464cc9b7d68866c3969a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.195323 14767 raft_consensus.cc:2804] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:26.195632 14767 raft_consensus.cc:493] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:26.195899 14767 raft_consensus.cc:3060] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.200853 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5a41643bad84006819fec986657e4ab" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:26.202116 14542 leader_election.cc:343] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:26.204952 14767 raft_consensus.cc:515] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.209702 14703 tablet_bootstrap.cc:654] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.232882 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "479d7fe06a284234aef420e3842f0eab" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:26.233464 14607 raft_consensus.cc:3060] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.241005 14607 raft_consensus.cc:2468] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:26.242004 14382 leader_election.cc:304] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:26.242761 14778 raft_consensus.cc:2804] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:26.248260 14778 raft_consensus.cc:697] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:26.249187 14778 consensus_queue.cc:237] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.252805 14703 tablet_bootstrap.cc:492] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:26.253168 14703 ts_tablet_manager.cc:1403] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.009s
I20251211 08:57:26.255261 14703 raft_consensus.cc:359] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.256067 14703 raft_consensus.cc:385] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.256297 14703 raft_consensus.cc:740] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.266134 14703 consensus_queue.cc:260] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.271721 14714 tablet_bootstrap.cc:492] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:26.271859 14703 ts_tablet_manager.cc:1434] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.018s user 0.003s sys 0.003s
I20251211 08:57:26.274969 14714 ts_tablet_manager.cc:1403] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.326s user 0.013s sys 0.001s
I20251211 08:57:26.276861 14714 raft_consensus.cc:359] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.278877 14714 raft_consensus.cc:385] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.282147 14703 tablet_bootstrap.cc:492] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:26.282378 14714 raft_consensus.cc:740] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.283087 14714 consensus_queue.cc:260] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.279582 14767 leader_election.cc:290] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:26.132815 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 1853070f81c34b52b8c74a7402e6dad3 (DEFAULT_TABLE table=default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.286481 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1853070f81c34b52b8c74a7402e6dad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.288039 14714 ts_tablet_manager.cc:1434] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20251211 08:57:26.288758 14714 tablet_bootstrap.cc:492] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:26.231591 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "479d7fe06a284234aef420e3842f0eab" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:26.291131 14703 tablet_bootstrap.cc:654] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.291052 14296 raft_consensus.cc:3060] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.298508 14714 tablet_bootstrap.cc:654] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.298386 14296 raft_consensus.cc:2468] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:26.306216 14703 tablet_bootstrap.cc:492] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:26.306576 14703 ts_tablet_manager.cc:1403] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.000s
I20251211 08:57:26.310078 14703 raft_consensus.cc:359] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.310719 14703 raft_consensus.cc:385] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.311111 14703 raft_consensus.cc:740] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
W20251211 08:57:26.311686 14544 leader_election.cc:343] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:26.312243 14544 leader_election.cc:304] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:26.313380 14773 raft_consensus.cc:2749] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:26.319321 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_bab313cc214e46fb8e94ad2b35c2d938"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:26.320703 14714 tablet_bootstrap.cc:492] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
W20251211 08:57:26.320919 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bab313cc214e46fb8e94ad2b35c2d938 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:26.321043 14714 ts_tablet_manager.cc:1403] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.000s
I20251211 08:57:26.322068 14703 consensus_queue.cc:260] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.323339 14714 raft_consensus.cc:359] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.323964 14714 raft_consensus.cc:385] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.324183 14714 raft_consensus.cc:740] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.323848 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_1ab8b3b3713f428293902db0aa8cedb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:26.325434 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ab8b3b3713f428293902db0aa8cedb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:26.326733 14703 ts_tablet_manager.cc:1434] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20251211 08:57:26.327008 14714 consensus_queue.cc:260] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.329749 14703 tablet_bootstrap.cc:492] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:26.330515 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_bf028d96f1394d5991707cb71858d809"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:26.332227 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf028d96f1394d5991707cb71858d809 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:26.335515 14703 tablet_bootstrap.cc:654] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.335577 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_1fe18f2fd26847cca06517f3a216c641" } new_table_name: "default.Table_c76caa256ffd4898a64d730f6583004f" modify_external_catalogs: true
I20251211 08:57:26.336409 14714 ts_tablet_manager.cc:1434] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20251211 08:57:26.337191 14714 tablet_bootstrap.cc:492] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:26.351251 14714 tablet_bootstrap.cc:654] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.360702 14778 raft_consensus.cc:493] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:26.361184 14778 raft_consensus.cc:515] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.363128 14778 leader_election.cc:290] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:26.368422 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 53fc34c41afe44678b8d6dcf79a31f8b (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]). Expected version 2 got 0
I20251211 08:57:26.369270 14714 tablet_bootstrap.cc:492] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:26.369640 14714 ts_tablet_manager.cc:1403] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.005s
I20251211 08:57:26.372080 14008 catalog_manager.cc:5654] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:26.372686 14714 raft_consensus.cc:359] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.373274 14714 raft_consensus.cc:385] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.373518 14714 raft_consensus.cc:740] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.374104 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet afdfd49bbb9742c69c66d14a35d6f3a0
I20251211 08:57:26.375710 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet afdfd49bbb9742c69c66d14a35d6f3a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:26.380165 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet afdfd49bbb9742c69c66d14a35d6f3a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:26.380810 14714 consensus_queue.cc:260] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.385030 14703 tablet_bootstrap.cc:492] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:26.385418 14703 ts_tablet_manager.cc:1403] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.056s user 0.021s sys 0.004s
I20251211 08:57:26.387575 14703 raft_consensus.cc:359] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.388113 14703 raft_consensus.cc:385] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.388322 14703 raft_consensus.cc:740] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.396216 14703 consensus_queue.cc:260] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.397634 14729 raft_consensus.cc:493] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:26.398059 14729 raft_consensus.cc:515] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.399158 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53fc34c41afe44678b8d6dcf79a31f8b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:26.399715 14607 raft_consensus.cc:2468] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:26.400784 14382 leader_election.cc:304] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:26.401654 14778 raft_consensus.cc:2804] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:26.402016 14778 raft_consensus.cc:493] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:26.402542 14778 raft_consensus.cc:3060] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.404451 14729 leader_election.cc:290] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:26.408278 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53fc34c41afe44678b8d6dcf79a31f8b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:26.409380 14385 leader_election.cc:343] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:26.416538 14703 ts_tablet_manager.cc:1434] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.031s user 0.005s sys 0.001s
I20251211 08:57:26.419101 14714 ts_tablet_manager.cc:1434] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.048s user 0.005s sys 0.000s
I20251211 08:57:26.420208 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet afdfd49bbb9742c69c66d14a35d6f3a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:26.427309 14717 tablet_replica.cc:333] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:26.428140 14717 raft_consensus.cc:2243] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:26.428675 14717 raft_consensus.cc:2272] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:26.431123 14717 ts_tablet_manager.cc:1916] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:26.431731 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "593df63696a04f85b00c301d93748e80" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:26.432364 14607 raft_consensus.cc:2468] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:26.433467 14230 leader_election.cc:304] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:26.434013 14714 tablet_bootstrap.cc:492] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:26.434726 14729 raft_consensus.cc:2804] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:26.435053 14729 raft_consensus.cc:493] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:26.435312 14729 raft_consensus.cc:3060] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.441408 14729 raft_consensus.cc:515] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.444931 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "593df63696a04f85b00c301d93748e80" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:26.445547 14607 raft_consensus.cc:3060] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.445888 14729 leader_election.cc:290] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:26.447238 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8e32b532fa954510a2efbc5305ae0aeb" } new_table_name: "default.Table_6b8cf2aad2794338bd00ec9e24500365" modify_external_catalogs: true
I20251211 08:57:26.447635 14703 tablet_bootstrap.cc:492] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:26.453857 14703 tablet_bootstrap.cc:654] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.459203 14607 raft_consensus.cc:2468] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:26.415302 14778 raft_consensus.cc:515] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.460253 14230 leader_election.cc:304] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:26.461088 14729 raft_consensus.cc:2804] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:26.465741 14717 ts_tablet_manager.cc:1929] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:26.466055 14717 log.cc:1199] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/afdfd49bbb9742c69c66d14a35d6f3a0
I20251211 08:57:26.466672 14717 ts_tablet_manager.cc:1950] T afdfd49bbb9742c69c66d14a35d6f3a0 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:26.472290 14722 tablet_bootstrap.cc:492] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:26.473543 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]) successfully deleted
I20251211 08:57:26.474444 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53fc34c41afe44678b8d6dcf79a31f8b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:26.475135 14607 raft_consensus.cc:3060] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.476733 14722 ts_tablet_manager.cc:1403] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.404s user 0.011s sys 0.000s
I20251211 08:57:26.481081 14722 raft_consensus.cc:359] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.482103 14729 raft_consensus.cc:697] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:26.483817 14729 consensus_queue.cc:237] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.492406 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53fc34c41afe44678b8d6dcf79a31f8b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:26.487708 14722 raft_consensus.cc:385] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251211 08:57:26.493929 14385 leader_election.cc:343] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:26.494110 14722 raft_consensus.cc:740] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.495118 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "593df63696a04f85b00c301d93748e80" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:26.495636 14449 raft_consensus.cc:2468] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:26.496493 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "593df63696a04f85b00c301d93748e80" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:26.496940 14448 raft_consensus.cc:3060] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.497537 14703 tablet_bootstrap.cc:492] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:26.497896 14703 ts_tablet_manager.cc:1403] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.006s
I20251211 08:57:26.498857 14778 leader_election.cc:290] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:26.499960 14703 raft_consensus.cc:359] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.500483 14703 raft_consensus.cc:385] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.500695 14703 raft_consensus.cc:740] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.501269 14703 consensus_queue.cc:260] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.503306 14703 ts_tablet_manager.cc:1434] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20251211 08:57:26.505100 14703 tablet_bootstrap.cc:492] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:26.504956 14448 raft_consensus.cc:2468] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:26.507836 14722 consensus_queue.cc:260] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.509678 14722 ts_tablet_manager.cc:1434] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.033s user 0.004s sys 0.000s
I20251211 08:57:26.510468 14722 tablet_bootstrap.cc:492] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:26.512614 14607 raft_consensus.cc:2468] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:26.514638 14382 leader_election.cc:304] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:26.518290 14778 raft_consensus.cc:2804] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:26.518946 14778 raft_consensus.cc:697] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:26.520841 14778 consensus_queue.cc:237] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.525910 14722 tablet_bootstrap.cc:654] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.528255 14714 tablet_bootstrap.cc:654] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.532688 14703 tablet_bootstrap.cc:654] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:26.532580 14382 consensus_peers.cc:597] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72789234f9e44b82ba438ce478d8372c. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:26.539402 14718 ts_tablet_manager.cc:1929] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:26.539858 14718 log.cc:1199] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/d7eff28d3edb45f7a9733fd6ea1d3d08
I20251211 08:57:26.540659 14718 ts_tablet_manager.cc:1950] T d7eff28d3edb45f7a9733fd6ea1d3d08 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:26.542917 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 22) took 1265 ms. Trace:
I20251211 08:57:26.557907 14005 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 36ebb707199c4b87b93a54ffec047d91
I20251211 08:57:26.568698 14703 tablet_bootstrap.cc:492] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:26.569070 14703 ts_tablet_manager.cc:1403] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.001s
I20251211 08:57:26.575265 14703 raft_consensus.cc:359] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.575870 14703 raft_consensus.cc:385] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.576102 14703 raft_consensus.cc:740] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.576808 14703 consensus_queue.cc:260] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.582259 14722 tablet_bootstrap.cc:492] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:26.582383 14720 ts_tablet_manager.cc:1929] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:26.582646 14722 ts_tablet_manager.cc:1403] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.003s
I20251211 08:57:26.582903 14720 log.cc:1199] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/72789234f9e44b82ba438ce478d8372c
I20251211 08:57:26.583626 14720 ts_tablet_manager.cc:1950] T 72789234f9e44b82ba438ce478d8372c P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:26.584811 14722 raft_consensus.cc:359] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.585397 14722 raft_consensus.cc:385] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.585644 14722 raft_consensus.cc:740] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.586313 14703 ts_tablet_manager.cc:1434] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20251211 08:57:26.587095 14703 tablet_bootstrap.cc:492] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:26.590706 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ead5cbe578404244a3c38f49d8dbc688" } modify_external_catalogs: true
I20251211 08:57:26.591266 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ead5cbe578404244a3c38f49d8dbc688" } modify_external_catalogs: true
I20251211 08:57:26.590874 14718 rpcz_store.cc:276] 1211 08:57:25.277040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:25.283413 (+ 6373us) service_pool.cc:225] Handling call
1211 08:57:25.817935 (+534522us) ts_tablet_manager.cc:1151] Deleting tablet d7eff28d3edb45f7a9733fd6ea1d3d08
1211 08:57:25.817987 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:26.261104 (+443117us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:26.539144 (+278040us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:26.542855 (+ 3711us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":446448}
I20251211 08:57:26.592655 14718 tablet_replica.cc:333] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:26.593502 14718 raft_consensus.cc:2243] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:26.594288 14718 pending_rounds.cc:70] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:26.594496 14718 pending_rounds.cc:77] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256352934883328 op_type: NO_OP noop_request { }
I20251211 08:57:26.598352 14722 consensus_queue.cc:260] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.599720 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet d7eff28d3edb45f7a9733fd6ea1d3d08 (table default.Table_8d6ceb6bcb7c48eeb7a85270f04731b2 [id=4be9f85009f34348ac6efa5d5bcb4b0d]) successfully deleted
I20251211 08:57:26.601181 14722 ts_tablet_manager.cc:1434] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20251211 08:57:26.601872 14722 tablet_bootstrap.cc:492] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:26.604300 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d4f6727c4464cc9b7d68866c3969a0f consensus state has no leader
I20251211 08:57:26.608466 14703 tablet_bootstrap.cc:654] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.611168 14722 tablet_bootstrap.cc:654] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.561857 14767 consensus_queue.cc:1048] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:26.614854 14803 consensus_queue.cc:1048] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:26.617326 14767 raft_consensus.cc:493] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:26.617697 14767 raft_consensus.cc:515] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.619454 14767 leader_election.cc:290] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:26.620129 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5a41643bad84006819fec986657e4ab" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:26.620649 14608 raft_consensus.cc:2468] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:26.621264 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5a41643bad84006819fec986657e4ab" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:26.621681 14382 leader_election.cc:304] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:26.621783 14299 raft_consensus.cc:2468] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:26.622390 14767 raft_consensus.cc:2804] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:26.622735 14767 raft_consensus.cc:493] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:26.627439 14718 raft_consensus.cc:2889] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:26.631098 14767 raft_consensus.cc:3060] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.631518 14718 raft_consensus.cc:2272] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:26.634296 14718 ts_tablet_manager.cc:1916] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:26.631969 14720 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 21) took 1152 ms. Trace:
I20251211 08:57:26.639532 14722 tablet_bootstrap.cc:492] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:26.639914 14722 ts_tablet_manager.cc:1403] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.005s
I20251211 08:57:26.638993 14720 rpcz_store.cc:276] 1211 08:57:25.479748 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:25.480359 (+ 611us) service_pool.cc:225] Handling call
1211 08:57:25.973468 (+493109us) ts_tablet_manager.cc:1151] Deleting tablet 72789234f9e44b82ba438ce478d8372c
1211 08:57:25.973507 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:26.430319 (+456812us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:26.582144 (+151825us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:26.631883 (+ 49739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":492600}
I20251211 08:57:26.642310 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 72789234f9e44b82ba438ce478d8372c (table default.Table_45109de5b01c40d98407cbe7c0de4fe4 [id=c110e62110d344be9fbae792cc0cfe62]) successfully deleted
W20251211 08:57:26.665668 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet bcdb31a588b84616844986e6101bb154 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcdb31a588b84616844986e6101bb154 already in progress: creating tablet
I20251211 08:57:26.673252 14703 tablet_bootstrap.cc:492] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:26.673780 14703 ts_tablet_manager.cc:1403] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.087s user 0.012s sys 0.000s
I20251211 08:57:26.675809 14703 raft_consensus.cc:359] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.676527 14703 raft_consensus.cc:385] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.676817 14703 raft_consensus.cc:740] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.677433 14703 consensus_queue.cc:260] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.637943 14767 raft_consensus.cc:515] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.643079 14714 tablet_bootstrap.cc:492] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
W20251211 08:57:26.660127 14382 consensus_peers.cc:597] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afdfd49bbb9742c69c66d14a35d6f3a0. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:26.678351 14714 ts_tablet_manager.cc:1403] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.244s user 0.009s sys 0.000s
I20251211 08:57:26.680438 14714 raft_consensus.cc:359] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:26.680742 13997 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 1853070f81c34b52b8c74a7402e6dad3 (table default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1853070f81c34b52b8c74a7402e6dad3 consensus state has no leader
I20251211 08:57:26.680958 14714 raft_consensus.cc:385] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.681213 14714 raft_consensus.cc:740] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.684165 14714 consensus_queue.cc:260] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.692000 14714 ts_tablet_manager.cc:1434] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20251211 08:57:26.695740 14714 tablet_bootstrap.cc:492] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:26.699697 14722 raft_consensus.cc:359] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.701902 14720 tablet_replica.cc:333] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:26.702986 14722 raft_consensus.cc:385] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.703269 14722 raft_consensus.cc:740] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.704176 14720 raft_consensus.cc:2243] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:26.704852 14720 raft_consensus.cc:2272] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:26.708581 14773 raft_consensus.cc:493] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:26.708981 14773 raft_consensus.cc:515] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.712491 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b60d99c2b4f442da4a4d9c5190f5797" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:26.712534 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b60d99c2b4f442da4a4d9c5190f5797" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:26.714085 14542 leader_election.cc:343] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:26.714946 14544 leader_election.cc:343] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:26.715364 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5a41643bad84006819fec986657e4ab" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:26.715896 14608 raft_consensus.cc:3060] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.715380 14544 leader_election.cc:304] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:26.710700 14773 leader_election.cc:290] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:26.719383 14773 raft_consensus.cc:2749] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:26.720638 14714 tablet_bootstrap.cc:654] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.724445 14608 raft_consensus.cc:2468] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:26.724874 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5a41643bad84006819fec986657e4ab" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:26.725369 14299 raft_consensus.cc:3060] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.731389 14722 consensus_queue.cc:260] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.733489 14299 raft_consensus.cc:2468] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:26.725526 14703 ts_tablet_manager.cc:1434] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.051s user 0.004s sys 0.000s
I20251211 08:57:26.734589 14385 leader_election.cc:304] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:26.736398 14729 raft_consensus.cc:493] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:26.736665 14803 raft_consensus.cc:2804] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:26.736829 14729 raft_consensus.cc:515] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.738442 14729 leader_election.cc:290] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:26.741561 14722 ts_tablet_manager.cc:1434] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.101s user 0.001s sys 0.004s
I20251211 08:57:26.748656 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f112603316047289459698ec28b54d4" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:26.749240 14449 raft_consensus.cc:2468] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:26.749337 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet d7eff28d3edb45f7a9733fd6ea1d3d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:26.736886 14767 leader_election.cc:290] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:26.762930 14703 tablet_bootstrap.cc:492] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:26.764694 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b0b4b0150de34800823b57578db1e8bb" } modify_external_catalogs: true
I20251211 08:57:26.766456 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b0b4b0150de34800823b57578db1e8bb" } modify_external_catalogs: true
I20251211 08:57:26.765727 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a7ade9d81bfc4498a43e0b5ad6941ac2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:26.768833 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7ade9d81bfc4498a43e0b5ad6941ac2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:26.771145 14714 tablet_bootstrap.cc:492] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:26.771546 14714 ts_tablet_manager.cc:1403] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.000s
I20251211 08:57:26.778956 14703 tablet_bootstrap.cc:654] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:26.780083 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 65160.7us
I20251211 08:57:26.781975 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f112603316047289459698ec28b54d4" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:26.782491 14608 raft_consensus.cc:2468] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:26.743402 14720 ts_tablet_manager.cc:1916] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:26.784488 14722 tablet_bootstrap.cc:492] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:26.784760 14714 raft_consensus.cc:359] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.785315 14714 raft_consensus.cc:385] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.785534 14714 raft_consensus.cc:740] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.752993 14231 leader_election.cc:304] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:26.789142 14714 consensus_queue.cc:260] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.792374 14803 raft_consensus.cc:697] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:26.792958 14714 ts_tablet_manager.cc:1434] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.021s user 0.004s sys 0.001s
I20251211 08:57:26.792958 14729 raft_consensus.cc:2804] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:26.793280 14729 raft_consensus.cc:493] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:26.793593 14729 raft_consensus.cc:3060] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.793150 14803 consensus_queue.cc:237] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:26.793795 14714 tablet_bootstrap.cc:492] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:26.794628 14703 tablet_bootstrap.cc:492] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:26.794978 14703 ts_tablet_manager.cc:1403] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.008s
I20251211 08:57:26.797223 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 8f112603316047289459698ec28b54d4 (table default.Table_b0b4b0150de34800823b57578db1e8bb [id=202989bd9772430baafe83f28d3debe4]). Expected version 1 got 0
W20251211 08:57:26.798820 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 593df63696a04f85b00c301d93748e80 consensus state has no leader
I20251211 08:57:26.806485 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet e5a41643bad84006819fec986657e4ab
I20251211 08:57:26.809278 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet e5a41643bad84006819fec986657e4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:26.809907 14717 tablet_replica.cc:333] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:26.810549 14717 raft_consensus.cc:2243] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:26.810961 14717 raft_consensus.cc:2272] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:26.812479 14717 ts_tablet_manager.cc:1916] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:26.815526 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet e5a41643bad84006819fec986657e4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:26.816989 14720 ts_tablet_manager.cc:1929] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:26.817366 14714 tablet_bootstrap.cc:654] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.819060 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_bf028d96f1394d5991707cb71858d809" } new_table_name: "default.Table_4b521cb9e71f4e11a462261413eb85b4" modify_external_catalogs: true
I20251211 08:57:26.810619 14729 raft_consensus.cc:515] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.818208 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet e5a41643bad84006819fec986657e4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:26.834048 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f112603316047289459698ec28b54d4" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:26.834869 14449 raft_consensus.cc:3060] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.834635 14703 raft_consensus.cc:359] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.835717 14703 raft_consensus.cc:385] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.835999 14703 raft_consensus.cc:740] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.841802 14449 raft_consensus.cc:2468] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:26.842877 14720 log.cc:1199] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/afdfd49bbb9742c69c66d14a35d6f3a0
I20251211 08:57:26.843703 14720 ts_tablet_manager.cc:1950] T afdfd49bbb9742c69c66d14a35d6f3a0 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:26.844458 14714 tablet_bootstrap.cc:492] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:26.844810 14714 ts_tablet_manager.cc:1403] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.003s
I20251211 08:57:26.846021 14720 tablet_replica.cc:333] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:26.846630 14720 raft_consensus.cc:2243] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:26.846861 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 8f112603316047289459698ec28b54d4
I20251211 08:57:26.846729 14714 raft_consensus.cc:359] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.847122 14720 raft_consensus.cc:2272] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:26.847278 14714 raft_consensus.cc:385] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:26.847474 14714 raft_consensus.cc:740] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:26.847954 14714 consensus_queue.cc:260] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.849936 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f112603316047289459698ec28b54d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:26.850978 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f112603316047289459698ec28b54d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:26.854658 14720 ts_tablet_manager.cc:1916] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:26.855764 14714 ts_tablet_manager.cc:1434] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20251211 08:57:26.855916 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet bcdb31a588b84616844986e6101bb154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:26.856549 14714 tablet_bootstrap.cc:492] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:26.858695 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]) successfully deleted
I20251211 08:57:26.859961 14703 consensus_queue.cc:260] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.861490 14718 ts_tablet_manager.cc:1929] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:26.861799 14718 log.cc:1199] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/72789234f9e44b82ba438ce478d8372c
I20251211 08:57:26.867609 14703 ts_tablet_manager.cc:1434] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.072s user 0.004s sys 0.000s
I20251211 08:57:26.871284 14703 tablet_bootstrap.cc:492] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:26.874035 14231 leader_election.cc:304] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:26.874926 14729 leader_election.cc:290] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:26.875347 14729 raft_consensus.cc:2804] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:26.875689 14729 raft_consensus.cc:697] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:26.876329 14729 consensus_queue.cc:237] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.881060 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f112603316047289459698ec28b54d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:26.885102 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f112603316047289459698ec28b54d4" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:26.885861 14608 raft_consensus.cc:3060] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.887846 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ba237fb232b24588ac35a6455f9476b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:26.889621 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba237fb232b24588ac35a6455f9476b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:26.891075 14828 raft_consensus.cc:493] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:26.891743 14828 raft_consensus.cc:515] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.893559 14828 leader_election.cc:290] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:26.895241 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5191443773b415198687c55c4517692 has no consensus state
I20251211 08:57:26.898226 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bcdb31a588b84616844986e6101bb154" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:26.899500 14544 leader_election.cc:343] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:26.900207 14722 tablet_bootstrap.cc:654] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.903353 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_86772d5552714163a6a0ffc236521623"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:26.904917 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86772d5552714163a6a0ffc236521623 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:26.907459 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f647a2c75e7c46939a62573dea122fef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:26.908380 14608 raft_consensus.cc:2468] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
W20251211 08:57:26.909024 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f647a2c75e7c46939a62573dea122fef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:26.921381 14703 tablet_bootstrap.cc:654] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.923624 14717 ts_tablet_manager.cc:1929] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:26.926254 14720 ts_tablet_manager.cc:1929] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:26.927402 14717 log.cc:1199] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/e5a41643bad84006819fec986657e4ab
I20251211 08:57:26.928033 14717 ts_tablet_manager.cc:1950] T e5a41643bad84006819fec986657e4ab P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:26.932212 14007 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 1853070f81c34b52b8c74a7402e6dad3 (table default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4]). Expected version 1 got 0
I20251211 08:57:26.937778 14588 tablet_service.cc:1505] Processing CreateTablet for tablet a9524aa1503549bd820918d1c025aa2e (DEFAULT_TABLE table=default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.938634 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9524aa1503549bd820918d1c025aa2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.946367 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 53fc34c41afe44678b8d6dcf79a31f8b (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]). Expected version 2 got 0
I20251211 08:57:26.947253 14013 catalog_manager.cc:5654] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:26.948626 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 593df63696a04f85b00c301d93748e80 (table default.Table_6b8cf2aad2794338bd00ec9e24500365 [id=ab93470ba8f54070be718c240d49aa13]). Expected version 1 got 0
I20251211 08:57:26.951936 14428 tablet_service.cc:1505] Processing CreateTablet for tablet a9524aa1503549bd820918d1c025aa2e (DEFAULT_TABLE table=default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.952948 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9524aa1503549bd820918d1c025aa2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.960301 14714 tablet_bootstrap.cc:654] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:26.961651 14425 tablet_service.cc:1505] Processing CreateTablet for tablet a0f9c321e673433498a9059dd2e2efe6 (DEFAULT_TABLE table=default.Table_6c1b92755bba45abbc6750537eabadb5 [id=649471fc3b2d4487b83e781f99ce77f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.962525 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0f9c321e673433498a9059dd2e2efe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.962663 14587 tablet_service.cc:1505] Processing CreateTablet for tablet a0f9c321e673433498a9059dd2e2efe6 (DEFAULT_TABLE table=default.Table_6c1b92755bba45abbc6750537eabadb5 [id=649471fc3b2d4487b83e781f99ce77f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.963651 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0f9c321e673433498a9059dd2e2efe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.971127 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bcdb31a588b84616844986e6101bb154" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:26.971817 14449 raft_consensus.cc:2468] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:26.974117 14542 leader_election.cc:304] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:26.974933 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet e5a41643bad84006819fec986657e4ab (table default.Table_ead5cbe578404244a3c38f49d8dbc688 [id=5ecbf67720c544f1bfed70017354edfe]) successfully deleted
I20251211 08:57:26.976007 14828 raft_consensus.cc:2804] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:26.976312 14828 raft_consensus.cc:493] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:26.976610 14828 raft_consensus.cc:3060] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.981858 14717 tablet_replica.cc:333] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:26.985139 14717 raft_consensus.cc:2243] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:26.985702 14717 raft_consensus.cc:2272] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:26.986495 14828 raft_consensus.cc:515] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.986917 14718 ts_tablet_manager.cc:1950] T 72789234f9e44b82ba438ce478d8372c P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:26.989213 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 23) took 1513 ms. Trace:
I20251211 08:57:26.989972 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bcdb31a588b84616844986e6101bb154" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:26.989938 14718 rpcz_store.cc:276] 1211 08:57:25.475568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:25.475732 (+ 164us) service_pool.cc:225] Handling call
1211 08:57:26.592490 (+1116758us) ts_tablet_manager.cc:1151] Deleting tablet 72789234f9e44b82ba438ce478d8372c
1211 08:57:26.592534 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:26.757894 (+165360us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:26.861299 (+103405us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:26.989153 (+127854us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1116233}
I20251211 08:57:26.990557 14449 raft_consensus.cc:3060] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:26.991242 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 72789234f9e44b82ba438ce478d8372c (table default.Table_45109de5b01c40d98407cbe7c0de4fe4 [id=c110e62110d344be9fbae792cc0cfe62]) successfully deleted
I20251211 08:57:26.993672 14718 tablet_replica.cc:333] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:26.994520 14717 ts_tablet_manager.cc:1916] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:26.996829 14773 raft_consensus.cc:493] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:26.997215 14773 raft_consensus.cc:515] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:26.932814 14275 tablet_service.cc:1505] Processing CreateTablet for tablet a9524aa1503549bd820918d1c025aa2e (DEFAULT_TABLE table=default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.998401 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9524aa1503549bd820918d1c025aa2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:26.999583 14773 leader_election.cc:290] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:26.945847 14278 tablet_service.cc:1505] Processing CreateTablet for tablet a0f9c321e673433498a9059dd2e2efe6 (DEFAULT_TABLE table=default.Table_6c1b92755bba45abbc6750537eabadb5 [id=649471fc3b2d4487b83e781f99ce77f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:26.989372 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bcdb31a588b84616844986e6101bb154" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:27.001941 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a2dcb9ac62e4aac885d638f4e2b072e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:27.003086 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0f9c321e673433498a9059dd2e2efe6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:27.004052 14544 leader_election.cc:343] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.004328 14714 tablet_bootstrap.cc:492] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:27.004699 14714 ts_tablet_manager.cc:1403] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.148s user 0.008s sys 0.002s
W20251211 08:57:27.004690 14544 leader_election.cc:343] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.005729 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a2dcb9ac62e4aac885d638f4e2b072e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:27.006208 14448 raft_consensus.cc:2468] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:27.007205 14542 leader_election.cc:304] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:27.007499 14714 raft_consensus.cc:359] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.008351 14773 raft_consensus.cc:2804] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:27.008600 14714 raft_consensus.cc:385] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.008772 14773 raft_consensus.cc:493] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:27.008852 14714 raft_consensus.cc:740] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.009181 14773 raft_consensus.cc:3060] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:27.012399 14714 consensus_queue.cc:260] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.014364 14714 ts_tablet_manager.cc:1434] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251211 08:57:27.015194 14714 tablet_bootstrap.cc:492] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:27.015455 14449 raft_consensus.cc:2468] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:27.021881 14718 raft_consensus.cc:2243] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:27.024369 14718 raft_consensus.cc:2272] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:27.025622 14717 ts_tablet_manager.cc:1929] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:27.027946 14714 tablet_bootstrap.cc:654] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.029631 14718 ts_tablet_manager.cc:1916] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:27.030604 14542 leader_election.cc:304] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:27.030328 14717 log.cc:1199] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/8f112603316047289459698ec28b54d4
I20251211 08:57:27.032353 14717 ts_tablet_manager.cc:1950] T 8f112603316047289459698ec28b54d4 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:27.033746 14828 leader_election.cc:290] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:27.034626 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 8f112603316047289459698ec28b54d4 (table default.Table_b0b4b0150de34800823b57578db1e8bb [id=202989bd9772430baafe83f28d3debe4]) successfully deleted
I20251211 08:57:27.035480 14717 tablet_replica.cc:333] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:27.036135 14717 raft_consensus.cc:2243] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:27.036542 14717 raft_consensus.cc:2272] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:27.039975 14714 tablet_bootstrap.cc:492] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:27.040383 14714 ts_tablet_manager.cc:1403] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.000s
I20251211 08:57:27.042218 14717 ts_tablet_manager.cc:1916] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:27.042495 14714 raft_consensus.cc:359] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.043346 14714 raft_consensus.cc:385] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.043751 14714 raft_consensus.cc:740] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.043906 14720 log.cc:1199] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/e5a41643bad84006819fec986657e4ab
I20251211 08:57:27.044620 14720 ts_tablet_manager.cc:1950] T e5a41643bad84006819fec986657e4ab P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:27.044512 14714 consensus_queue.cc:260] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.017352 14773 raft_consensus.cc:515] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.048353 14714 ts_tablet_manager.cc:1434] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20251211 08:57:27.048058 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7c70be2e41de4127a511a479fdb6ad7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:27.049894 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c70be2e41de4127a511a479fdb6ad7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:27.050985 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a2dcb9ac62e4aac885d638f4e2b072e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:27.051311 14714 tablet_bootstrap.cc:492] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:27.052271 14544 leader_election.cc:343] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.054147 14773 leader_election.cc:290] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:27.054642 14011 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_cbeef8d5f95540b9a51b10a0a18a2dce" } modify_external_catalogs: true
I20251211 08:57:27.055230 14011 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_cbeef8d5f95540b9a51b10a0a18a2dce" } modify_external_catalogs: true
I20251211 08:57:27.055929 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a2dcb9ac62e4aac885d638f4e2b072e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:27.056646 14449 raft_consensus.cc:3060] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:27.058645 14714 tablet_bootstrap.cc:654] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.059998 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_1bef0864cfa04e77a665ae61d8cccbe3" } new_table_name: "default.Table_b6bdd942603d47ef84a718099c2e7c40" modify_external_catalogs: true
I20251211 08:57:27.067514 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_bc568f3abe9d41fc84ca9e7c0bf3b28d" } modify_external_catalogs: true
I20251211 08:57:27.068305 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_bc568f3abe9d41fc84ca9e7c0bf3b28d" } modify_external_catalogs: true
I20251211 08:57:27.085528 14449 raft_consensus.cc:2468] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:27.101763 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_6c1b92755bba45abbc6750537eabadb5" } new_table_name: "default.Table_d429fb0e546d4f3d886de3a04af241cf" modify_external_catalogs: true
I20251211 08:57:27.103888 14542 leader_election.cc:304] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:27.105731 14773 raft_consensus.cc:2804] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:27.110432 14840 raft_consensus.cc:493] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:27.110948 14840 raft_consensus.cc:515] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.106362 14773 raft_consensus.cc:697] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:27.113270 14773 consensus_queue.cc:237] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.115427 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad903727f79a4180adf047c92d981c7e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:27.115689 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad903727f79a4180adf047c92d981c7e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:27.116703 14544 leader_election.cc:343] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:27.117751 14542 leader_election.cc:343] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.118997 14542 leader_election.cc:304] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:27.120087 14828 raft_consensus.cc:2749] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:27.121162 14722 tablet_bootstrap.cc:492] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:27.116972 14840 leader_election.cc:290] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:27.121701 14722 ts_tablet_manager.cc:1403] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.337s user 0.001s sys 0.007s
I20251211 08:57:27.124639 14722 raft_consensus.cc:359] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.125427 14722 raft_consensus.cc:385] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.125738 14722 raft_consensus.cc:740] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.136440 14720 tablet_replica.cc:333] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:27.136389 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet e5a41643bad84006819fec986657e4ab (table default.Table_ead5cbe578404244a3c38f49d8dbc688 [id=5ecbf67720c544f1bfed70017354edfe]) successfully deleted
I20251211 08:57:27.137290 14720 raft_consensus.cc:2243] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:27.138193 14720 pending_rounds.cc:70] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
I20251211 08:57:27.098100 14717 ts_tablet_manager.cc:1929] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:27.159134 14717 log.cc:1199] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/bcdb31a588b84616844986e6101bb154
I20251211 08:57:27.159871 14717 ts_tablet_manager.cc:1950] T bcdb31a588b84616844986e6101bb154 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:27.162077 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet bcdb31a588b84616844986e6101bb154 (table default.Table_8709763a5bb141899fbb13faa5e4a24d [id=5a9a6d684a1a496595f5bb9ddfca3f46]) successfully deleted
I20251211 08:57:27.163484 14718 ts_tablet_manager.cc:1929] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:27.164119 14718 log.cc:1199] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/bcdb31a588b84616844986e6101bb154
I20251211 08:57:27.164842 14718 ts_tablet_manager.cc:1950] T bcdb31a588b84616844986e6101bb154 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:27.168362 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 28) took 1346 ms. Trace:
I20251211 08:57:27.169528 14718 rpcz_store.cc:276] 1211 08:57:25.822196 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:25.822330 (+ 134us) service_pool.cc:225] Handling call
1211 08:57:26.993520 (+1171190us) ts_tablet_manager.cc:1151] Deleting tablet bcdb31a588b84616844986e6101bb154
1211 08:57:26.993567 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:27.024075 (+ 30508us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b54001202d4. stack: 00007efeecdf4083 00007efeecdf3f80 00007efeec969574 0000000000508744 00000000005086d9 00007efef3e3b5a1 00007efef3ed9fe2 00007efef42fc2ec 00007efef4f5331a 00007efef4f83f80 00007efef4f6a54d 00007efef4f6a4f9 00007efef4f6a489 00007efef4f6a451 00007efef4f6967d 00000000005aa2b4
1211 08:57:27.087668 (+ 63593us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:27.163250 (+ 75582us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:27.168308 (+ 5058us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":60575488,"tablet-delete.queue_time_us":1169605}
I20251211 08:57:27.175263 14718 tablet_replica.cc:333] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:27.175303 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet bcdb31a588b84616844986e6101bb154 (table default.Table_8709763a5bb141899fbb13faa5e4a24d [id=5a9a6d684a1a496595f5bb9ddfca3f46]) successfully deleted
I20251211 08:57:27.176298 14718 raft_consensus.cc:2243] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:27.177460 14718 raft_consensus.cc:2272] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:27.179828 14718 ts_tablet_manager.cc:1916] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:27.186498 14703 tablet_bootstrap.cc:492] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:27.189122 14703 ts_tablet_manager.cc:1403] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.318s user 0.004s sys 0.006s
I20251211 08:57:27.197449 14722 consensus_queue.cc:260] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.200119 14722 ts_tablet_manager.cc:1434] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.078s user 0.007s sys 0.001s
I20251211 08:57:27.211306 14011 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 7b60d99c2b4f442da4a4d9c5190f5797
I20251211 08:57:27.213263 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b60d99c2b4f442da4a4d9c5190f5797 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:27.213654 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b60d99c2b4f442da4a4d9c5190f5797 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:27.214299 14717 tablet_replica.cc:333] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:27.214999 14717 raft_consensus.cc:2243] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:27.215387 14717 raft_consensus.cc:2272] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:27.216946 14717 ts_tablet_manager.cc:1916] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:27.222355 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_70c7872ad47d44eda242c89515fae159"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:27.224184 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70c7872ad47d44eda242c89515fae159 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:27.225654 14773 raft_consensus.cc:493] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:27.226065 14773 raft_consensus.cc:515] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.227800 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b225e6a7b22c43e1ae401c96cf7eaad8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:27.228227 14773 leader_election.cc:290] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:27.229266 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b225e6a7b22c43e1ae401c96cf7eaad8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:27.230043 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292b641df22f478d8849ff6148ecf951" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:27.230641 14449 raft_consensus.cc:2468] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:27.231539 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292b641df22f478d8849ff6148ecf951" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:27.231904 14542 leader_election.cc:304] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:27.232728 14773 raft_consensus.cc:2804] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251211 08:57:27.233014 14544 leader_election.cc:343] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.233139 14773 raft_consensus.cc:493] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:27.233582 14773 raft_consensus.cc:3060] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:27.239902 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b60d99c2b4f442da4a4d9c5190f5797 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:27.138473 14720 pending_rounds.cc:77] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256358412222464 op_type: NO_OP noop_request { }
I20251211 08:57:27.243022 14720 raft_consensus.cc:2889] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:27.243489 14720 raft_consensus.cc:2272] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:27.245867 14720 ts_tablet_manager.cc:1916] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:27.242471 14773 raft_consensus.cc:515] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.263461 14703 raft_consensus.cc:359] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.264225 14703 raft_consensus.cc:385] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.264462 14703 raft_consensus.cc:740] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.265726 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292b641df22f478d8849ff6148ecf951" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:27.266280 14449 raft_consensus.cc:3060] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:27.267524 14714 tablet_bootstrap.cc:492] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:27.267906 14714 ts_tablet_manager.cc:1403] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.219s user 0.033s sys 0.047s
I20251211 08:57:27.270056 14714 raft_consensus.cc:359] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.274042 14449 raft_consensus.cc:2468] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:27.275758 14542 leader_election.cc:304] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:27.276933 14840 raft_consensus.cc:2804] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:27.277520 14840 raft_consensus.cc:697] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:27.278512 14840 consensus_queue.cc:237] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.283895 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet d6c52b08ccaf49c48fe909b72a5da45f
I20251211 08:57:27.284453 14773 leader_election.cc:290] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:27.286690 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c52b08ccaf49c48fe909b72a5da45f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:27.288293 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c52b08ccaf49c48fe909b72a5da45f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:27.295763 14718 ts_tablet_manager.cc:1929] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:27.296705 14717 ts_tablet_manager.cc:1929] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:27.299664 14718 log.cc:1199] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/afdfd49bbb9742c69c66d14a35d6f3a0
I20251211 08:57:27.300319 14718 ts_tablet_manager.cc:1950] T afdfd49bbb9742c69c66d14a35d6f3a0 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:27.300590 14717 log.cc:1199] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/7b60d99c2b4f442da4a4d9c5190f5797
I20251211 08:57:27.301115 14717 ts_tablet_manager.cc:1950] T 7b60d99c2b4f442da4a4d9c5190f5797 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:27.304823 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet afdfd49bbb9742c69c66d14a35d6f3a0 (table default.Table_07efa184e6744e9cbbd8e869edb7a3b2 [id=1370e8a22c3f447485b6b019c0f195b9]) successfully deleted
I20251211 08:57:27.305559 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f0b7cb6abf9b49fd8b0051bb2eb9e55e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:27.307307 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0b7cb6abf9b49fd8b0051bb2eb9e55e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:27.308952 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 7b60d99c2b4f442da4a4d9c5190f5797 (table default.Table_cbeef8d5f95540b9a51b10a0a18a2dce [id=71569e18b5d347598f45f7e38c073515]) successfully deleted
W20251211 08:57:27.309816 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet d7eff28d3edb45f7a9733fd6ea1d3d08 because the tablet was not found. No further retry: Not found: Tablet not found: d7eff28d3edb45f7a9733fd6ea1d3d08
I20251211 08:57:27.310441 14718 tablet_replica.cc:333] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:27.256042 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet bcdb31a588b84616844986e6101bb154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:25 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:27.259972 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet e5a41643bad84006819fec986657e4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:27.260936 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292b641df22f478d8849ff6148ecf951" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:27.266247 14729 raft_consensus.cc:493] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:27.311970 14729 raft_consensus.cc:515] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:27.313687 14729 leader_election.cc:290] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:27.315219 14544 leader_election.cc:343] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.315347 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53fc34c41afe44678b8d6dcf79a31f8b" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:27.287879 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c52b08ccaf49c48fe909b72a5da45f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:27.316007 14608 raft_consensus.cc:2393] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:27.299070 14813 consensus_queue.cc:1048] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:27.316335 14703 consensus_queue.cc:260] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.317142 14773 raft_consensus.cc:493] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:27.317507 14773 raft_consensus.cc:515] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.318279 14703 ts_tablet_manager.cc:1434] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.129s user 0.004s sys 0.000s
I20251211 08:57:27.319018 14703 tablet_bootstrap.cc:492] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:27.320971 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53fc34c41afe44678b8d6dcf79a31f8b" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:27.322278 14231 leader_election.cc:304] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:27.323253 14813 raft_consensus.cc:3060] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:27.330662 14813 raft_consensus.cc:2749] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:27.303032 14722 tablet_bootstrap.cc:492] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:27.332818 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c181adf9db574ab39432b2f9b14fe4d7" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:27.334087 14544 leader_election.cc:343] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.336406 14773 leader_election.cc:290] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:27.338598 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c181adf9db574ab39432b2f9b14fe4d7" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:27.339646 14542 leader_election.cc:343] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.340001 14542 leader_election.cc:304] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:27.340827 14773 raft_consensus.cc:2749] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:27.341823 14722 tablet_bootstrap.cc:654] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.270608 14714 raft_consensus.cc:385] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.343269 14714 raft_consensus.cc:740] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.344713 14717 tablet_replica.cc:333] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:27.345397 14717 raft_consensus.cc:2243] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:27.346421 14718 raft_consensus.cc:2243] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:27.347016 14718 pending_rounds.cc:70] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:27.347234 14718 pending_rounds.cc:77] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256358072283136 op_type: NO_OP noop_request { }
I20251211 08:57:27.347503 14718 raft_consensus.cc:2889] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:27.347764 14718 raft_consensus.cc:2272] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:27.348632 14714 consensus_queue.cc:260] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.353780 14717 raft_consensus.cc:2272] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:27.355521 14718 ts_tablet_manager.cc:1916] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:27.356808 14714 ts_tablet_manager.cc:1434] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.089s user 0.006s sys 0.000s
I20251211 08:57:27.366005 14722 tablet_bootstrap.cc:492] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:27.366457 14722 ts_tablet_manager.cc:1403] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.166s user 0.009s sys 0.002s
I20251211 08:57:27.368559 14722 raft_consensus.cc:359] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.384019 14813 consensus_queue.cc:1048] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:27.378901 14722 raft_consensus.cc:385] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.385931 14722 raft_consensus.cc:740] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.387168 14722 consensus_queue.cc:260] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.393333 14722 ts_tablet_manager.cc:1434] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20251211 08:57:27.394680 14722 tablet_bootstrap.cc:492] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:27.396047 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0f9c321e673433498a9059dd2e2efe6 consensus state has no leader
I20251211 08:57:27.403424 14803 consensus_queue.cc:1048] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:27.406430 14722 tablet_bootstrap.cc:654] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.410272 14703 tablet_bootstrap.cc:654] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.412326 14718 ts_tablet_manager.cc:1929] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:27.412674 14718 log.cc:1199] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/e5a41643bad84006819fec986657e4ab
I20251211 08:57:27.413411 14718 ts_tablet_manager.cc:1950] T e5a41643bad84006819fec986657e4ab P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:27.425547 14722 tablet_bootstrap.cc:492] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:27.425915 14722 ts_tablet_manager.cc:1403] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.000s
I20251211 08:57:27.433369 14803 consensus_queue.cc:1048] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:27.435427 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet e5a41643bad84006819fec986657e4ab (table default.Table_ead5cbe578404244a3c38f49d8dbc688 [id=5ecbf67720c544f1bfed70017354edfe]) successfully deleted
I20251211 08:57:27.434688 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a7ade9d81bfc4498a43e0b5ad6941ac2" } modify_external_catalogs: true
I20251211 08:57:27.436195 14722 raft_consensus.cc:359] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.436378 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a7ade9d81bfc4498a43e0b5ad6941ac2" } modify_external_catalogs: true
I20251211 08:57:27.436760 14722 raft_consensus.cc:385] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.436990 14722 raft_consensus.cc:740] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.437894 14718 tablet_replica.cc:333] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:27.438472 14718 raft_consensus.cc:2243] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:27.438841 14718 raft_consensus.cc:2272] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:27.440251 14718 ts_tablet_manager.cc:1916] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:27.447507 14722 consensus_queue.cc:260] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.451200 14722 ts_tablet_manager.cc:1434] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20251211 08:57:27.451982 14722 tablet_bootstrap.cc:492] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:27.460795 14803 consensus_queue.cc:1048] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:27.483047 14850 raft_consensus.cc:493] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:27.483543 14850 raft_consensus.cc:515] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.487167 14714 tablet_bootstrap.cc:492] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:27.489830 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4b521cb9e71f4e11a462261413eb85b4" } modify_external_catalogs: true
I20251211 08:57:27.490347 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4b521cb9e71f4e11a462261413eb85b4" } modify_external_catalogs: true
W20251211 08:57:27.492961 14231 leader_election.cc:343] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: bcdb31a588b84616844986e6101bb154
I20251211 08:57:27.493281 14720 ts_tablet_manager.cc:1929] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:27.493607 14720 log.cc:1199] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/8f112603316047289459698ec28b54d4
I20251211 08:57:27.499128 14703 tablet_bootstrap.cc:492] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:27.499545 14703 ts_tablet_manager.cc:1403] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.181s user 0.008s sys 0.000s
I20251211 08:57:27.501174 14717 ts_tablet_manager.cc:1916] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:27.501886 14703 raft_consensus.cc:359] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.498274 14722 tablet_bootstrap.cc:654] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.501504 14850 leader_election.cc:290] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:27.503504 14230 leader_election.cc:343] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: bcdb31a588b84616844986e6101bb154
I20251211 08:57:27.504101 14230 leader_election.cc:304] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:27.505574 14850 raft_consensus.cc:2749] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:27.505919 14720 ts_tablet_manager.cc:1950] T 8f112603316047289459698ec28b54d4 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:27.506860 14703 raft_consensus.cc:385] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.507126 14703 raft_consensus.cc:740] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.507710 14703 consensus_queue.cc:260] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.508085 14718 ts_tablet_manager.cc:1929] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:27.508427 14718 log.cc:1199] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/8f112603316047289459698ec28b54d4
I20251211 08:57:27.508977 14718 ts_tablet_manager.cc:1950] T 8f112603316047289459698ec28b54d4 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:27.509658 14703 ts_tablet_manager.cc:1434] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20251211 08:57:27.510298 14703 tablet_bootstrap.cc:492] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:27.512075 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 8f112603316047289459698ec28b54d4 (table default.Table_b0b4b0150de34800823b57578db1e8bb [id=202989bd9772430baafe83f28d3debe4]) successfully deleted
I20251211 08:57:27.512392 14720 tablet_replica.cc:333] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:27.512998 14720 raft_consensus.cc:2243] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:27.513044 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 8f112603316047289459698ec28b54d4 (table default.Table_b0b4b0150de34800823b57578db1e8bb [id=202989bd9772430baafe83f28d3debe4]) successfully deleted
I20251211 08:57:27.513377 14720 raft_consensus.cc:2272] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:27.513700 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 7b60d99c2b4f442da4a4d9c5190f5797 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b60d99c2b4f442da4a4d9c5190f5797 already in progress: creating tablet
I20251211 08:57:27.514657 14718 tablet_replica.cc:333] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
W20251211 08:57:27.513731 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 7b60d99c2b4f442da4a4d9c5190f5797 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b60d99c2b4f442da4a4d9c5190f5797 already in progress: creating tablet
I20251211 08:57:27.515486 14718 raft_consensus.cc:2243] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:27.529419 14007 catalog_manager.cc:5654] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:27.529811 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 45f3395a34cf4fdd8e1898b08d123c8a (DEFAULT_TABLE table=default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.531407 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f3395a34cf4fdd8e1898b08d123c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.535005 14007 catalog_manager.cc:5654] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:27.536731 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 45f3395a34cf4fdd8e1898b08d123c8a (DEFAULT_TABLE table=default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.537804 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f3395a34cf4fdd8e1898b08d123c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.533305 14010 catalog_manager.cc:5654] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
W20251211 08:57:27.540824 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1a2dcb9ac62e4aac885d638f4e2b072e consensus state has no leader
I20251211 08:57:27.542747 14588 tablet_service.cc:1505] Processing CreateTablet for tablet c5191443773b415198687c55c4517692 (DEFAULT_TABLE table=default.Table_4b521cb9e71f4e11a462261413eb85b4 [id=118053de49694cac881977b65f70b182]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.543738 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5191443773b415198687c55c4517692. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.545925 14277 tablet_service.cc:1505] Processing CreateTablet for tablet c5191443773b415198687c55c4517692 (DEFAULT_TABLE table=default.Table_4b521cb9e71f4e11a462261413eb85b4 [id=118053de49694cac881977b65f70b182]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.546826 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5191443773b415198687c55c4517692. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.548590 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 53fc34c41afe44678b8d6dcf79a31f8b (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]). Expected version 2 got 0
I20251211 08:57:27.551424 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 593df63696a04f85b00c301d93748e80 (table default.Table_6b8cf2aad2794338bd00ec9e24500365 [id=ab93470ba8f54070be718c240d49aa13]). Expected version 1 got 0
I20251211 08:57:27.567759 14773 raft_consensus.cc:493] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:27.568127 14773 raft_consensus.cc:515] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.570127 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1853070f81c34b52b8c74a7402e6dad3" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:27.570065 14278 tablet_service.cc:1505] Processing CreateTablet for tablet 426e26859b9c4966b8ec8ba3eab53a43 (DEFAULT_TABLE table=default.Table_6169dbdabeee4eaeab82230228af27b9 [id=69dd3ef114034ac7a3d60b8a0fde70b6]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:27.571261 14544 leader_election.cc:343] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.571578 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426e26859b9c4966b8ec8ba3eab53a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.571835 14773 leader_election.cc:290] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:27.572448 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 426e26859b9c4966b8ec8ba3eab53a43 (DEFAULT_TABLE table=default.Table_6169dbdabeee4eaeab82230228af27b9 [id=69dd3ef114034ac7a3d60b8a0fde70b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.573448 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426e26859b9c4966b8ec8ba3eab53a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.574025 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 38655ee4996e41cc8f87b5248264613d (DEFAULT_TABLE table=default.Table_a7ade9d81bfc4498a43e0b5ad6941ac2 [id=c533026c25a54aa885eb483d4120f720]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.574939 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38655ee4996e41cc8f87b5248264613d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.579181 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 38655ee4996e41cc8f87b5248264613d (DEFAULT_TABLE table=default.Table_a7ade9d81bfc4498a43e0b5ad6941ac2 [id=c533026c25a54aa885eb483d4120f720]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.580054 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38655ee4996e41cc8f87b5248264613d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.589824 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4c07a2c494ed47d29a486da1035027e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:27.592067 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c07a2c494ed47d29a486da1035027e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:27.603480 14584 tablet_service.cc:1505] Processing CreateTablet for tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 (DEFAULT_TABLE table=default.Table_1ab8b3b3713f428293902db0aa8cedb6 [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.604548 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab1ee5cf4eb4e11a72f5ad1de8c3856. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.612870 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_6169dbdabeee4eaeab82230228af27b9" } new_table_name: "default.Table_a144e458c8554442a03170f402a13d01" modify_external_catalogs: true
I20251211 08:57:27.621444 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 38655ee4996e41cc8f87b5248264613d
I20251211 08:57:27.532071 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 45f3395a34cf4fdd8e1898b08d123c8a (DEFAULT_TABLE table=default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.625376 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f3395a34cf4fdd8e1898b08d123c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.626143 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet 38655ee4996e41cc8f87b5248264613d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:27.542873 14428 tablet_service.cc:1505] Processing CreateTablet for tablet c5191443773b415198687c55c4517692 (DEFAULT_TABLE table=default.Table_4b521cb9e71f4e11a462261413eb85b4 [id=118053de49694cac881977b65f70b182]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.629137 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5191443773b415198687c55c4517692. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.630911 14866 consensus_queue.cc:1048] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:27.632604 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 38655ee4996e41cc8f87b5248264613d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:27.635095 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_d1d406b278304b5ca6e6cc3b63e7fd36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:27.636672 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1d406b278304b5ca6e6cc3b63e7fd36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:27.545862 14803 raft_consensus.cc:493] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:27.580528 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 426e26859b9c4966b8ec8ba3eab53a43 (DEFAULT_TABLE table=default.Table_6169dbdabeee4eaeab82230228af27b9 [id=69dd3ef114034ac7a3d60b8a0fde70b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.637620 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426e26859b9c4966b8ec8ba3eab53a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.639900 14868 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:27.587366 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1853070f81c34b52b8c74a7402e6dad3" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:27.647158 14717 ts_tablet_manager.cc:1929] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251211 08:57:27.647467 14717 log.cc:1199] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/d6c52b08ccaf49c48fe909b72a5da45f
W20251211 08:57:27.647807 14542 leader_election.cc:343] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.648195 14542 leader_election.cc:304] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:27.649077 14773 raft_consensus.cc:2749] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:27.648046 14717 ts_tablet_manager.cc:1950] T d6c52b08ccaf49c48fe909b72a5da45f P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:27.594446 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 38655ee4996e41cc8f87b5248264613d (DEFAULT_TABLE table=default.Table_a7ade9d81bfc4498a43e0b5ad6941ac2 [id=c533026c25a54aa885eb483d4120f720]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.650223 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38655ee4996e41cc8f87b5248264613d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.653263 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet d6c52b08ccaf49c48fe909b72a5da45f (table default.Table_bc568f3abe9d41fc84ca9e7c0bf3b28d [id=704c4d83c2464876a16db15b76817d50]) successfully deleted
I20251211 08:57:27.654858 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_94b52b113a9f45b69bff699f4d2bdd0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:27.655375 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 38655ee4996e41cc8f87b5248264613d because the tablet was not found. No further retry: Not found: Tablet not found: 38655ee4996e41cc8f87b5248264613d
I20251211 08:57:27.656424 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 38655ee4996e41cc8f87b5248264613d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:27.656438 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94b52b113a9f45b69bff699f4d2bdd0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:27.598076 14714 tablet_bootstrap.cc:654] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.606078 14426 tablet_service.cc:1505] Processing CreateTablet for tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 (DEFAULT_TABLE table=default.Table_1ab8b3b3713f428293902db0aa8cedb6 [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.659484 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab1ee5cf4eb4e11a72f5ad1de8c3856. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.659216 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 559f9d428b5841708999725b5b859660 (DEFAULT_TABLE table=default.Table_bab313cc214e46fb8e94ad2b35c2d938 [id=3f3f000cd93e44df9f084873ba4b168b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.660460 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559f9d428b5841708999725b5b859660. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.665807 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet c5191443773b415198687c55c4517692
I20251211 08:57:27.654989 14276 tablet_service.cc:1505] Processing CreateTablet for tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 (DEFAULT_TABLE table=default.Table_1ab8b3b3713f428293902db0aa8cedb6 [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.668404 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab1ee5cf4eb4e11a72f5ad1de8c3856. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.670918 14803 raft_consensus.cc:515] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.671445 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet c5191443773b415198687c55c4517692 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:27.673847 14803 leader_election.cc:290] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:27.674417 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4f6727c4464cc9b7d68866c3969a0f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:27.674973 14607 raft_consensus.cc:2468] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:27.675773 14382 leader_election.cc:304] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:27.676527 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet c5191443773b415198687c55c4517692 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:27.677052 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet c5191443773b415198687c55c4517692 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:27.678998 14720 ts_tablet_manager.cc:1916] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:27.679065 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4f6727c4464cc9b7d68866c3969a0f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:27.680052 14385 leader_election.cc:343] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.680824 14803 raft_consensus.cc:2804] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:27.681105 14803 raft_consensus.cc:493] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:27.681372 14803 raft_consensus.cc:3060] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:27.691578 14429 tablet_service.cc:1505] Processing CreateTablet for tablet 559f9d428b5841708999725b5b859660 (DEFAULT_TABLE table=default.Table_bab313cc214e46fb8e94ad2b35c2d938 [id=3f3f000cd93e44df9f084873ba4b168b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.692641 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559f9d428b5841708999725b5b859660. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.697239 14703 tablet_bootstrap.cc:654] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.699457 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_fe31f77461b643beb43b390d2e857446"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:27.701010 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe31f77461b643beb43b390d2e857446 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:27.702695 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet c5191443773b415198687c55c4517692 because the tablet was not found. No further retry: Not found: Tablet not found: c5191443773b415198687c55c4517692
W20251211 08:57:27.703118 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet a0f9c321e673433498a9059dd2e2efe6 (table default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a0f9c321e673433498a9059dd2e2efe6 consensus state has no leader
I20251211 08:57:27.707882 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 559f9d428b5841708999725b5b859660 (DEFAULT_TABLE table=default.Table_bab313cc214e46fb8e94ad2b35c2d938 [id=3f3f000cd93e44df9f084873ba4b168b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:27.708249 14718 raft_consensus.cc:2272] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:27.708904 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559f9d428b5841708999725b5b859660. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:27.709904 14718 ts_tablet_manager.cc:1916] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:27.728029 14722 tablet_bootstrap.cc:492] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:27.728466 14722 ts_tablet_manager.cc:1403] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.277s user 0.010s sys 0.003s
I20251211 08:57:27.739086 14722 raft_consensus.cc:359] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.739789 14722 raft_consensus.cc:385] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.740073 14722 raft_consensus.cc:740] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.740725 14722 consensus_queue.cc:260] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.743515 14722 ts_tablet_manager.cc:1434] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20251211 08:57:27.744314 14722 tablet_bootstrap.cc:492] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:27.772240 14714 tablet_bootstrap.cc:492] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:27.772652 14714 ts_tablet_manager.cc:1403] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.286s user 0.008s sys 0.005s
I20251211 08:57:27.776708 14803 raft_consensus.cc:515] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.783620 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_6b8cf2aad2794338bd00ec9e24500365" } new_table_name: "default.Table_116c9891f83d47438f27b42312aa4cce" modify_external_catalogs: true
I20251211 08:57:27.800958 14722 tablet_bootstrap.cc:654] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.815990 14861 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:27.827793 14703 tablet_bootstrap.cc:492] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:27.828168 14703 ts_tablet_manager.cc:1403] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.318s user 0.006s sys 0.002s
I20251211 08:57:27.830142 14703 raft_consensus.cc:359] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.834168 14011 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ea8ceaf3dcc7478ea8aceec86870f8dd" } modify_external_catalogs: true
I20251211 08:57:27.834899 14011 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ea8ceaf3dcc7478ea8aceec86870f8dd" } modify_external_catalogs: true
I20251211 08:57:27.830754 14703 raft_consensus.cc:385] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.859191 14703 raft_consensus.cc:740] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.859889 14703 consensus_queue.cc:260] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.833669 14850 raft_consensus.cc:493] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:27.864205 14850 raft_consensus.cc:515] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.866144 14722 tablet_bootstrap.cc:492] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:27.866067 14850 leader_election.cc:290] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:27.866514 14722 ts_tablet_manager.cc:1403] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.122s user 0.008s sys 0.000s
I20251211 08:57:27.867609 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a2dcb9ac62e4aac885d638f4e2b072e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:27.868566 14722 raft_consensus.cc:359] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.869097 14722 raft_consensus.cc:385] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.869323 14722 raft_consensus.cc:740] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.870978 14703 ts_tablet_manager.cc:1434] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.043s user 0.007s sys 0.000s
I20251211 08:57:27.873162 14720 ts_tablet_manager.cc:1929] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:27.873543 14720 log.cc:1199] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/bcdb31a588b84616844986e6101bb154
I20251211 08:57:27.874233 14720 ts_tablet_manager.cc:1950] T bcdb31a588b84616844986e6101bb154 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:27.884039 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet bcdb31a588b84616844986e6101bb154 (table default.Table_8709763a5bb141899fbb13faa5e4a24d [id=5a9a6d684a1a496595f5bb9ddfca3f46]) successfully deleted
I20251211 08:57:27.885043 14720 tablet_replica.cc:333] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:27.885833 14720 raft_consensus.cc:2243] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:27.886178 14703 tablet_bootstrap.cc:492] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:27.889858 14720 raft_consensus.cc:2272] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:27.891830 14703 tablet_bootstrap.cc:654] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.892818 14720 ts_tablet_manager.cc:1916] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:27.895615 14803 leader_election.cc:290] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:27.896785 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4f6727c4464cc9b7d68866c3969a0f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:27.896952 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4f6727c4464cc9b7d68866c3969a0f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:27.897491 14608 raft_consensus.cc:3060] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:27.897967 14385 leader_election.cc:343] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:27.774680 14714 raft_consensus.cc:359] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.899578 14714 raft_consensus.cc:385] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.899830 14714 raft_consensus.cc:740] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.900381 14714 consensus_queue.cc:260] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.902285 14714 ts_tablet_manager.cc:1434] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.129s user 0.005s sys 0.000s
I20251211 08:57:27.904134 14608 raft_consensus.cc:2468] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:27.905054 14382 leader_election.cc:304] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:27.906196 14864 raft_consensus.cc:2804] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:27.906540 14864 raft_consensus.cc:697] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:27.907183 14864 consensus_queue.cc:237] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.912220 14722 consensus_queue.cc:260] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.915257 14722 ts_tablet_manager.cc:1434] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20251211 08:57:27.916472 14722 tablet_bootstrap.cc:492] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:27.931048 14703 tablet_bootstrap.cc:492] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:27.931548 14703 ts_tablet_manager.cc:1403] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.000s
I20251211 08:57:27.935232 14714 tablet_bootstrap.cc:492] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:27.939915 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 426e26859b9c4966b8ec8ba3eab53a43 consensus state has no leader
I20251211 08:57:27.945523 14720 ts_tablet_manager.cc:1929] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251211 08:57:27.945914 14720 log.cc:1199] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/d6c52b08ccaf49c48fe909b72a5da45f
I20251211 08:57:27.946609 14720 ts_tablet_manager.cc:1950] T d6c52b08ccaf49c48fe909b72a5da45f P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:27.949066 14299 raft_consensus.cc:3060] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:27.956687 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet d6c52b08ccaf49c48fe909b72a5da45f (table default.Table_bc568f3abe9d41fc84ca9e7c0bf3b28d [id=704c4d83c2464876a16db15b76817d50]) successfully deleted
W20251211 08:57:27.957283 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet e5a41643bad84006819fec986657e4ab because the tablet was not found. No further retry: Not found: Tablet not found: e5a41643bad84006819fec986657e4ab
W20251211 08:57:27.957777 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 38655ee4996e41cc8f87b5248264613d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38655ee4996e41cc8f87b5248264613d already in progress: creating tablet
W20251211 08:57:27.958236 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet c5191443773b415198687c55c4517692 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5191443773b415198687c55c4517692 already in progress: creating tablet
I20251211 08:57:27.967614 14703 raft_consensus.cc:359] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.968452 14703 raft_consensus.cc:385] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:27.968762 14703 raft_consensus.cc:740] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:27.969990 14722 tablet_bootstrap.cc:654] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.970345 14703 consensus_queue.cc:260] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:27.877257 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a2dcb9ac62e4aac885d638f4e2b072e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:27.857651 14871 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:27.876433 14862 raft_consensus.cc:493] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:27.986021 14714 tablet_bootstrap.cc:654] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:27.986527 14449 raft_consensus.cc:2393] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate c5e624bcfd154cd586ef1dedb9940063 in this term.
I20251211 08:57:27.977349 14299 raft_consensus.cc:1275] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Refusing update from remote peer c5e624bcfd154cd586ef1dedb9940063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:28.001303 14448 raft_consensus.cc:1275] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Refusing update from remote peer c5e624bcfd154cd586ef1dedb9940063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:28.001988 14231 leader_election.cc:304] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:28.006389 14722 tablet_bootstrap.cc:492] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:28.006706 14722 ts_tablet_manager.cc:1403] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.090s user 0.008s sys 0.000s
I20251211 08:57:28.007236 14850 raft_consensus.cc:2749] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:28.008564 14722 raft_consensus.cc:359] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.009084 14722 raft_consensus.cc:385] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.009302 14722 raft_consensus.cc:740] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.010596 14773 consensus_queue.cc:1048] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:28.012933 14880 consensus_queue.cc:1048] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:28.016985 14703 ts_tablet_manager.cc:1434] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.085s user 0.006s sys 0.000s
I20251211 08:57:27.988574 14862 raft_consensus.cc:515] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.023967 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad903727f79a4180adf047c92d981c7e" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:28.024557 14296 raft_consensus.cc:2468] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:28.025676 14385 leader_election.cc:304] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:28.026757 14864 raft_consensus.cc:2804] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:28.027263 14864 raft_consensus.cc:493] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:28.027539 14864 raft_consensus.cc:3060] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.009968 14722 consensus_queue.cc:260] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.036194 14722 ts_tablet_manager.cc:1434] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20251211 08:57:28.036932 14722 tablet_bootstrap.cc:492] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:28.036525 14864 raft_consensus.cc:515] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.040534 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7c70be2e41de4127a511a479fdb6ad7e" } modify_external_catalogs: true
I20251211 08:57:28.041112 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7c70be2e41de4127a511a479fdb6ad7e" } modify_external_catalogs: true
I20251211 08:57:28.041213 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad903727f79a4180adf047c92d981c7e" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:28.041726 14296 raft_consensus.cc:3060] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.047633 14703 tablet_bootstrap.cc:492] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:28.049664 14722 tablet_bootstrap.cc:654] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.049808 14296 raft_consensus.cc:2468] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:28.051755 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad903727f79a4180adf047c92d981c7e" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:28.052387 14608 raft_consensus.cc:2468] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:28.054724 14862 leader_election.cc:290] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:28.057098 14385 leader_election.cc:304] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:28.058319 14866 raft_consensus.cc:2804] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:28.058838 14866 raft_consensus.cc:697] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:28.059636 14866 consensus_queue.cc:237] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.074728 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad903727f79a4180adf047c92d981c7e" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:28.075294 14608 raft_consensus.cc:3060] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.075776 14703 tablet_bootstrap.cc:654] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.083926 14608 raft_consensus.cc:2468] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:28.085893 14880 raft_consensus.cc:493] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:28.086643 14880 raft_consensus.cc:515] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.090224 14850 raft_consensus.cc:493] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:28.090797 14850 raft_consensus.cc:515] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.092754 14718 ts_tablet_manager.cc:1929] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251211 08:57:28.092949 14850 leader_election.cc:290] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:28.093124 14718 log.cc:1199] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/d6c52b08ccaf49c48fe909b72a5da45f
I20251211 08:57:28.093822 14718 ts_tablet_manager.cc:1950] T d6c52b08ccaf49c48fe909b72a5da45f P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:28.094960 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292b641df22f478d8849ff6148ecf951" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:28.095846 14449 raft_consensus.cc:2393] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate c5e624bcfd154cd586ef1dedb9940063 in this term.
I20251211 08:57:28.099908 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet d6c52b08ccaf49c48fe909b72a5da45f (table default.Table_bc568f3abe9d41fc84ca9e7c0bf3b28d [id=704c4d83c2464876a16db15b76817d50]) successfully deleted
W20251211 08:57:28.102308 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 38655ee4996e41cc8f87b5248264613d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38655ee4996e41cc8f87b5248264613d already in progress: creating tablet
W20251211 08:57:28.104130 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet c5191443773b415198687c55c4517692 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5191443773b415198687c55c4517692 already in progress: creating tablet
I20251211 08:57:28.106514 14722 tablet_bootstrap.cc:492] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:28.106925 14722 ts_tablet_manager.cc:1403] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.070s user 0.008s sys 0.007s
I20251211 08:57:28.108729 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9524aa1503549bd820918d1c025aa2e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:28.109922 14542 leader_election.cc:343] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:28.110648 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9524aa1503549bd820918d1c025aa2e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:28.111729 14703 tablet_bootstrap.cc:492] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
W20251211 08:57:28.111971 14544 leader_election.cc:343] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:28.112113 14703 ts_tablet_manager.cc:1403] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.000s
I20251211 08:57:28.112366 14544 leader_election.cc:304] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:28.113155 14881 raft_consensus.cc:2749] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:28.115183 14703 raft_consensus.cc:359] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.115573 14881 raft_consensus.cc:493] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:28.115813 14703 raft_consensus.cc:385] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.116140 14703 raft_consensus.cc:740] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.115962 14881 raft_consensus.cc:515] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.116763 14703 consensus_queue.cc:260] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.118422 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0f9c321e673433498a9059dd2e2efe6" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:28.118754 14881 leader_election.cc:290] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:28.120034 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0f9c321e673433498a9059dd2e2efe6" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:28.122793 14544 leader_election.cc:343] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:28.109347 14722 raft_consensus.cc:359] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.124607 14722 raft_consensus.cc:385] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.124868 14722 raft_consensus.cc:740] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.126135 14722 consensus_queue.cc:260] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.125638 14703 ts_tablet_manager.cc:1434] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
W20251211 08:57:28.126667 14542 leader_election.cc:343] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:28.127202 14542 leader_election.cc:304] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:28.128135 14881 raft_consensus.cc:2749] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:28.129954 14703 tablet_bootstrap.cc:492] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:28.130254 14714 tablet_bootstrap.cc:492] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:28.130650 14714 ts_tablet_manager.cc:1403] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.196s user 0.007s sys 0.002s
I20251211 08:57:28.133288 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292b641df22f478d8849ff6148ecf951" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:28.139074 14230 leader_election.cc:304] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:28.140174 14850 raft_consensus.cc:3060] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.146302 14850 raft_consensus.cc:2749] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:28.146782 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2670ab0571da4c8f8ccbeee370ba3914"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:28.148564 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2670ab0571da4c8f8ccbeee370ba3914 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:28.137029 14880 leader_election.cc:290] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:28.151199 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_70c7872ad47d44eda242c89515fae159" } new_table_name: "default.Table_582685f00d3f4077bb184ec676e2ad02" modify_external_catalogs: true
I20251211 08:57:28.148599 14864 leader_election.cc:290] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:28.159648 14881 consensus_queue.cc:1048] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:28.179706 14861 tablet.cc:1722] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:28.181113 14868 tablet.cc:1722] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:28.202314 14722 ts_tablet_manager.cc:1434] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20251211 08:57:28.203644 14722 tablet_bootstrap.cc:492] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:28.218179 14722 tablet_bootstrap.cc:654] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.220793 14714 raft_consensus.cc:359] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.221455 14714 raft_consensus.cc:385] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.221711 14714 raft_consensus.cc:740] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.224615 14703 tablet_bootstrap.cc:654] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.225404 14871 tablet.cc:1722] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:28.233798 14881 consensus_queue.cc:1048] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:28.233884 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4d13dd70ed244e54861bb2debdf39715"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:28.235721 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d13dd70ed244e54861bb2debdf39715 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:28.243487 14894 tablet.cc:1722] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:28.256419 14861 tablet.cc:1722] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:28.267714 14714 consensus_queue.cc:260] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.272699 14714 ts_tablet_manager.cc:1434] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.141s user 0.004s sys 0.000s
I20251211 08:57:28.277572 14714 tablet_bootstrap.cc:492] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:28.287822 14714 tablet_bootstrap.cc:654] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.296053 14868 tablet.cc:1722] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:28.300132 14714 tablet_bootstrap.cc:492] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:28.300577 14714 ts_tablet_manager.cc:1403] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.002s
I20251211 08:57:28.302788 14714 raft_consensus.cc:359] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.303385 14714 raft_consensus.cc:385] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.303613 14714 raft_consensus.cc:740] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.304193 14714 consensus_queue.cc:260] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.312404 14714 ts_tablet_manager.cc:1434] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20251211 08:57:28.313223 14714 tablet_bootstrap.cc:492] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:28.320794 14714 tablet_bootstrap.cc:654] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.334750 14714 tablet_bootstrap.cc:492] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:28.335160 14714 ts_tablet_manager.cc:1403] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.004s
I20251211 08:57:28.337128 14714 raft_consensus.cc:359] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.337705 14714 raft_consensus.cc:385] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.337936 14714 raft_consensus.cc:740] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.338578 14714 consensus_queue.cc:260] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.340719 14703 tablet_bootstrap.cc:492] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:28.341113 14703 ts_tablet_manager.cc:1403] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.211s user 0.005s sys 0.005s
I20251211 08:57:28.344044 14703 raft_consensus.cc:359] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.344534 14714 ts_tablet_manager.cc:1434] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.009s user 0.004s sys 0.002s
I20251211 08:57:28.344782 14703 raft_consensus.cc:385] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.345053 14703 raft_consensus.cc:740] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.345747 14703 consensus_queue.cc:260] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.347167 14714 tablet_bootstrap.cc:492] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:28.351260 14722 tablet_bootstrap.cc:492] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:28.351647 14722 ts_tablet_manager.cc:1403] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.148s user 0.023s sys 0.010s
I20251211 08:57:28.352540 14714 tablet_bootstrap.cc:654] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.353915 14722 raft_consensus.cc:359] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.354511 14722 raft_consensus.cc:385] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.363312 14722 raft_consensus.cc:740] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.367645 14714 tablet_bootstrap.cc:492] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:28.367986 14714 ts_tablet_manager.cc:1403] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.006s
I20251211 08:57:28.369840 14714 raft_consensus.cc:359] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.370396 14714 raft_consensus.cc:385] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.370625 14714 raft_consensus.cc:740] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.371317 14714 consensus_queue.cc:260] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.376847 14714 ts_tablet_manager.cc:1434] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.009s user 0.002s sys 0.002s
I20251211 08:57:28.378400 14714 tablet_bootstrap.cc:492] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:28.378445 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 7b9b338a3f544ccb813247250b0f0eb7 (DEFAULT_TABLE table=default.Table_f0b7cb6abf9b49fd8b0051bb2eb9e55e [id=85b5244918f44d04a067fe1aa8b35975]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.381373 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b9b338a3f544ccb813247250b0f0eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.382642 14005 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 53fc34c41afe44678b8d6dcf79a31f8b (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]). Expected version 2 got 0
I20251211 08:57:28.384644 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 7b9b338a3f544ccb813247250b0f0eb7 (DEFAULT_TABLE table=default.Table_f0b7cb6abf9b49fd8b0051bb2eb9e55e [id=85b5244918f44d04a067fe1aa8b35975]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.385516 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b9b338a3f544ccb813247250b0f0eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.386945 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 7b9b338a3f544ccb813247250b0f0eb7 (DEFAULT_TABLE table=default.Table_f0b7cb6abf9b49fd8b0051bb2eb9e55e [id=85b5244918f44d04a067fe1aa8b35975]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.387833 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b9b338a3f544ccb813247250b0f0eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.391798 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 12379255548f49dfad17f3840a0c86bb (DEFAULT_TABLE table=default.Table_b225e6a7b22c43e1ae401c96cf7eaad8 [id=e10eaa167f0a46c39674a5e2212f2226]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:28.392306 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00f79094ed8b4782a3a1488e41794f9b consensus state has no leader
I20251211 08:57:28.392822 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12379255548f49dfad17f3840a0c86bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.393312 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_e478355b49e940ddaa167200d689b1e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:28.364178 14722 consensus_queue.cc:260] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.396513 14722 ts_tablet_manager.cc:1434] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20251211 08:57:28.396694 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b091ae28b8ef4b35aaf2dbf0cac0f59d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:28.397291 14722 tablet_bootstrap.cc:492] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:28.402097 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 12379255548f49dfad17f3840a0c86bb (DEFAULT_TABLE table=default.Table_b225e6a7b22c43e1ae401c96cf7eaad8 [id=e10eaa167f0a46c39674a5e2212f2226]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.403149 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12379255548f49dfad17f3840a0c86bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.402567 14587 tablet_service.cc:1505] Processing CreateTablet for tablet e015d0fb9d6c4c80a510c05244b334e4 (DEFAULT_TABLE table=default.Table_f647a2c75e7c46939a62573dea122fef [id=779be79e440a45d7ac4f6afb05bed936]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.406390 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e015d0fb9d6c4c80a510c05244b334e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.406910 14703 ts_tablet_manager.cc:1434] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.066s user 0.005s sys 0.000s
I20251211 08:57:28.407407 14428 tablet_service.cc:1505] Processing CreateTablet for tablet e015d0fb9d6c4c80a510c05244b334e4 (DEFAULT_TABLE table=default.Table_f647a2c75e7c46939a62573dea122fef [id=779be79e440a45d7ac4f6afb05bed936]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.408257 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e015d0fb9d6c4c80a510c05244b334e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.412057 14426 tablet_service.cc:1505] Processing CreateTablet for tablet d235bade012540e1bb1c0dd871eff1bb (DEFAULT_TABLE table=default.Table_7c70be2e41de4127a511a479fdb6ad7e [id=5b6d7c4fd998498f91fe1594382caed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.413061 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d235bade012540e1bb1c0dd871eff1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.416054 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 12379255548f49dfad17f3840a0c86bb (DEFAULT_TABLE table=default.Table_b225e6a7b22c43e1ae401c96cf7eaad8 [id=e10eaa167f0a46c39674a5e2212f2226]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.417052 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12379255548f49dfad17f3840a0c86bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.419508 14274 tablet_service.cc:1505] Processing CreateTablet for tablet e015d0fb9d6c4c80a510c05244b334e4 (DEFAULT_TABLE table=default.Table_f647a2c75e7c46939a62573dea122fef [id=779be79e440a45d7ac4f6afb05bed936]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.420425 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e015d0fb9d6c4c80a510c05244b334e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.420316 14588 tablet_service.cc:1505] Processing CreateTablet for tablet d235bade012540e1bb1c0dd871eff1bb (DEFAULT_TABLE table=default.Table_7c70be2e41de4127a511a479fdb6ad7e [id=5b6d7c4fd998498f91fe1594382caed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.420598 14585 tablet_service.cc:1505] Processing CreateTablet for tablet d2c6080b3c1e4897bc1881e50f29f3a0 (DEFAULT_TABLE table=default.Table_86772d5552714163a6a0ffc236521623 [id=a5f4dfd6520045919362ac16b16d4acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.421191 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d235bade012540e1bb1c0dd871eff1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.421711 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c6080b3c1e4897bc1881e50f29f3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.423717 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 00f79094ed8b4782a3a1488e41794f9b (DEFAULT_TABLE table=default.Table_582685f00d3f4077bb184ec676e2ad02 [id=ade2f567017f4807ad913bb1d62948b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.424842 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f79094ed8b4782a3a1488e41794f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.427416 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 00f79094ed8b4782a3a1488e41794f9b (DEFAULT_TABLE table=default.Table_582685f00d3f4077bb184ec676e2ad02 [id=ade2f567017f4807ad913bb1d62948b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.428354 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f79094ed8b4782a3a1488e41794f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.429406 14862 raft_consensus.cc:493] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:28.429731 14429 tablet_service.cc:1505] Processing CreateTablet for tablet d2c6080b3c1e4897bc1881e50f29f3a0 (DEFAULT_TABLE table=default.Table_86772d5552714163a6a0ffc236521623 [id=a5f4dfd6520045919362ac16b16d4acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.429797 14862 raft_consensus.cc:515] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.431407 14277 tablet_service.cc:1505] Processing CreateTablet for tablet d235bade012540e1bb1c0dd871eff1bb (DEFAULT_TABLE table=default.Table_7c70be2e41de4127a511a479fdb6ad7e [id=5b6d7c4fd998498f91fe1594382caed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.431612 14862 leader_election.cc:290] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:28.432359 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d235bade012540e1bb1c0dd871eff1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.435945 14278 tablet_service.cc:1505] Processing CreateTablet for tablet d2c6080b3c1e4897bc1881e50f29f3a0 (DEFAULT_TABLE table=default.Table_86772d5552714163a6a0ffc236521623 [id=a5f4dfd6520045919362ac16b16d4acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.436292 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 00f79094ed8b4782a3a1488e41794f9b (DEFAULT_TABLE table=default.Table_582685f00d3f4077bb184ec676e2ad02 [id=ade2f567017f4807ad913bb1d62948b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:28.436935 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c6080b3c1e4897bc1881e50f29f3a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:28.439088 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b091ae28b8ef4b35aaf2dbf0cac0f59d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:28.439251 14703 tablet_bootstrap.cc:492] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:28.442462 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f79094ed8b4782a3a1488e41794f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.443073 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1853070f81c34b52b8c74a7402e6dad3" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:28.443107 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1853070f81c34b52b8c74a7402e6dad3" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:28.443696 14299 raft_consensus.cc:2468] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:28.443887 14608 raft_consensus.cc:2468] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:28.444870 14385 leader_election.cc:304] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:28.445109 14881 raft_consensus.cc:493] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:28.446004 14862 raft_consensus.cc:2804] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:28.446336 14862 raft_consensus.cc:493] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:28.446426 14722 tablet_bootstrap.cc:654] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.446635 14862 raft_consensus.cc:3060] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.445516 14881 raft_consensus.cc:515] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.430707 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c6080b3c1e4897bc1881e50f29f3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:28.450254 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45f3395a34cf4fdd8e1898b08d123c8a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:28.451458 14542 leader_election.cc:343] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:28.452144 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45f3395a34cf4fdd8e1898b08d123c8a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:28.453287 14544 leader_election.cc:343] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:28.453678 14544 leader_election.cc:304] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:28.454336 14893 raft_consensus.cc:2749] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:28.460423 14862 raft_consensus.cc:515] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.462783 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1853070f81c34b52b8c74a7402e6dad3" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:28.463321 14703 tablet_bootstrap.cc:654] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.463308 14299 raft_consensus.cc:3060] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:28.466944 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e478355b49e940ddaa167200d689b1e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:28.468981 14299 raft_consensus.cc:2468] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:28.469825 14385 leader_election.cc:304] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:28.470469 14864 raft_consensus.cc:2804] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:28.470927 14864 raft_consensus.cc:697] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:28.471518 14862 leader_election.cc:290] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:28.471706 14864 consensus_queue.cc:237] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.479063 14850 raft_consensus.cc:493] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:28.479416 14850 raft_consensus.cc:515] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.481103 14850 leader_election.cc:290] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:28.482333 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4f6727c4464cc9b7d68866c3969a0f" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:28.482758 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4f6727c4464cc9b7d68866c3969a0f" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:28.483485 14607 raft_consensus.cc:2393] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:28.484539 14230 leader_election.cc:304] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:28.485625 14850 raft_consensus.cc:3060] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.491986 14850 raft_consensus.cc:2749] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:28.495023 14893 raft_consensus.cc:493] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:28.495390 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1853070f81c34b52b8c74a7402e6dad3" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:28.495867 14608 raft_consensus.cc:3060] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.495606 14893 raft_consensus.cc:515] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.497231 14703 tablet_bootstrap.cc:492] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:28.497577 14703 ts_tablet_manager.cc:1403] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.059s user 0.010s sys 0.000s
I20251211 08:57:28.497416 14893 leader_election.cc:290] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:28.499122 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "426e26859b9c4966b8ec8ba3eab53a43" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:28.500257 14542 leader_election.cc:343] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:28.500939 14005 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 1a2dcb9ac62e4aac885d638f4e2b072e (table default.Table_b6bdd942603d47ef84a718099c2e7c40 [id=1673af33c2f846c7bf092303eff6e8ad]). Expected version 1 got 0
I20251211 08:57:28.501935 14005 catalog_manager.cc:5654] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:28.503041 14703 raft_consensus.cc:359] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.503516 14005 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 7d4f6727c4464cc9b7d68866c3969a0f (table default.Table_c76caa256ffd4898a64d730f6583004f [id=e6bc81a67ba949c69798c5049da505b5]). Expected version 1 got 0
I20251211 08:57:28.503669 14703 raft_consensus.cc:385] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.503937 14703 raft_consensus.cc:740] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.504485 14005 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 1853070f81c34b52b8c74a7402e6dad3 (table default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4]). Expected version 1 got 0
I20251211 08:57:28.454321 14881 leader_election.cc:290] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:28.504659 14703 consensus_queue.cc:260] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.510860 14703 ts_tablet_manager.cc:1434] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20251211 08:57:28.511713 14703 tablet_bootstrap.cc:492] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:28.528404 14703 tablet_bootstrap.cc:654] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.498811 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "426e26859b9c4966b8ec8ba3eab53a43" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:28.500172 14722 tablet_bootstrap.cc:492] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:28.529634 14722 ts_tablet_manager.cc:1403] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.133s user 0.007s sys 0.004s
W20251211 08:57:28.530258 14544 leader_election.cc:343] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:28.530674 14544 leader_election.cc:304] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:28.531678 14893 raft_consensus.cc:2749] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:28.531886 14722 raft_consensus.cc:359] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.532459 14722 raft_consensus.cc:385] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.532846 14722 raft_consensus.cc:740] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.533393 14722 consensus_queue.cc:260] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.536713 14722 ts_tablet_manager.cc:1434] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251211 08:57:28.538339 14722 tablet_bootstrap.cc:492] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:28.545466 14866 raft_consensus.cc:493] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:28.545894 14866 raft_consensus.cc:515] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.547222 14011 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet d582d092ce05451e9ef541c56de6e387
I20251211 08:57:28.547636 14866 leader_election.cc:290] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:28.549453 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c181adf9db574ab39432b2f9b14fe4d7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:28.549983 14299 raft_consensus.cc:2468] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:28.551188 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet d582d092ce05451e9ef541c56de6e387 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:28.551296 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c181adf9db574ab39432b2f9b14fe4d7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:28.551868 14607 raft_consensus.cc:2468] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:28.552467 14722 tablet_bootstrap.cc:654] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.552693 14385 leader_election.cc:304] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:28.553740 14862 raft_consensus.cc:2804] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:28.554057 14862 raft_consensus.cc:493] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:28.554179 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet d582d092ce05451e9ef541c56de6e387 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:28.554368 14862 raft_consensus.cc:3060] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.559563 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet d582d092ce05451e9ef541c56de6e387 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:28.560377 14718 tablet_replica.cc:333] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:28.561182 14718 raft_consensus.cc:2243] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:28.562004 14718 raft_consensus.cc:2272] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:28.565137 14718 ts_tablet_manager.cc:1916] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:28.574474 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet d235bade012540e1bb1c0dd871eff1bb
I20251211 08:57:28.576113 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet d235bade012540e1bb1c0dd871eff1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:28 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:28.578365 14582 tablet_service.cc:1552] Processing DeleteTablet for tablet d235bade012540e1bb1c0dd871eff1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:28 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:28.578400 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet d235bade012540e1bb1c0dd871eff1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:28 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:28.580003 14703 tablet_bootstrap.cc:492] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:28.580338 14703 ts_tablet_manager.cc:1403] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.069s user 0.014s sys 0.014s
I20251211 08:57:28.582286 14703 raft_consensus.cc:359] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.582870 14703 raft_consensus.cc:385] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.583114 14703 raft_consensus.cc:740] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.583652 14703 consensus_queue.cc:260] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.585822 14703 ts_tablet_manager.cc:1434] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20251211 08:57:28.586589 14703 tablet_bootstrap.cc:492] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:28.590052 14608 raft_consensus.cc:2468] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:28.592303 14703 tablet_bootstrap.cc:654] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.594671 14722 tablet_bootstrap.cc:492] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:28.595170 14722 ts_tablet_manager.cc:1403] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.057s user 0.008s sys 0.003s
I20251211 08:57:28.597314 14722 raft_consensus.cc:359] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.597847 14722 raft_consensus.cc:385] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.606962 14722 raft_consensus.cc:740] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.611011 14722 consensus_queue.cc:260] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.616212 14703 tablet_bootstrap.cc:492] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:28.616560 14703 ts_tablet_manager.cc:1403] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.002s
I20251211 08:57:28.617626 14722 ts_tablet_manager.cc:1434] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20251211 08:57:28.618335 14722 tablet_bootstrap.cc:492] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:28.618521 14703 raft_consensus.cc:359] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.619081 14703 raft_consensus.cc:385] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.619314 14703 raft_consensus.cc:740] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.619915 14703 consensus_queue.cc:260] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.622133 14703 ts_tablet_manager.cc:1434] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20251211 08:57:28.629715 14722 tablet_bootstrap.cc:654] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.645078 14905 tablet_replica.cc:333] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:28.646041 14905 raft_consensus.cc:2243] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:28.646543 14905 raft_consensus.cc:2272] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:28.650303 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_47c60a996b0642519864b928857b2c86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:28.653364 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47c60a996b0642519864b928857b2c86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:28.654330 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4e029d83ecc947918a83e94beb8871c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:28.656088 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e029d83ecc947918a83e94beb8871c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:28.651819 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b6bdd942603d47ef84a718099c2e7c40" } new_table_name: "default.Table_4550b341c9cc4c82b83b4daf817232bd" modify_external_catalogs: true
I20251211 08:57:28.666203 14010 catalog_manager.cc:5654] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:28.667855 14010 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 7d4f6727c4464cc9b7d68866c3969a0f (table default.Table_c76caa256ffd4898a64d730f6583004f [id=e6bc81a67ba949c69798c5049da505b5]). Expected version 1 got 0
I20251211 08:57:28.669277 14722 tablet_bootstrap.cc:492] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:28.669143 14010 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 53fc34c41afe44678b8d6dcf79a31f8b (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]). Expected version 2 got 0
I20251211 08:57:28.669605 14722 ts_tablet_manager.cc:1403] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.000s
I20251211 08:57:28.671559 14722 raft_consensus.cc:359] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.672108 14722 raft_consensus.cc:385] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:28.672335 14722 raft_consensus.cc:740] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:28.672966 14722 consensus_queue.cc:260] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.679358 14722 ts_tablet_manager.cc:1434] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20251211 08:57:28.690444 14718 ts_tablet_manager.cc:1929] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:28.690910 14718 log.cc:1199] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/d582d092ce05451e9ef541c56de6e387
I20251211 08:57:28.692063 14866 raft_consensus.cc:493] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:28.692503 14866 raft_consensus.cc:515] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.693178 14850 raft_consensus.cc:493] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:28.693585 14850 raft_consensus.cc:515] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.694247 14866 leader_election.cc:290] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:28.695093 14718 ts_tablet_manager.cc:1950] T d582d092ce05451e9ef541c56de6e387 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:28.695319 14850 leader_election.cc:290] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:28.715621 14862 raft_consensus.cc:515] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.723100 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9524aa1503549bd820918d1c025aa2e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:28.728750 14449 raft_consensus.cc:2468] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:28.730057 14231 leader_election.cc:304] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:28.731077 14850 raft_consensus.cc:2804] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:28.731539 14850 raft_consensus.cc:493] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:28.731920 14850 raft_consensus.cc:3060] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.735754 14862 leader_election.cc:290] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:28.737072 14382 leader_election.cc:343] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 7b60d99c2b4f442da4a4d9c5190f5797
I20251211 08:57:28.747045 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet d582d092ce05451e9ef541c56de6e387 (table default.Table_ea8ceaf3dcc7478ea8aceec86870f8dd [id=deaf44db4c1146a8ac640aa3ef778467]) successfully deleted
W20251211 08:57:28.747886 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet d235bade012540e1bb1c0dd871eff1bb because the tablet was not found. No further retry: Not found: Tablet not found: d235bade012540e1bb1c0dd871eff1bb
W20251211 08:57:28.752132 13997 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 00f79094ed8b4782a3a1488e41794f9b (table default.Table_582685f00d3f4077bb184ec676e2ad02 [id=ade2f567017f4807ad913bb1d62948b9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 00f79094ed8b4782a3a1488e41794f9b consensus state has no leader
I20251211 08:57:28.754812 14011 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b091ae28b8ef4b35aaf2dbf0cac0f59d" } modify_external_catalogs: true
I20251211 08:57:28.755550 14011 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b091ae28b8ef4b35aaf2dbf0cac0f59d" } modify_external_catalogs: true
I20251211 08:57:28.779402 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_1ac7df210845496091096027805f9fff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:28.781096 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ac7df210845496091096027805f9fff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:28.781311 14714 tablet_bootstrap.cc:654] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.696995 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9524aa1503549bd820918d1c025aa2e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:28.811973 14608 raft_consensus.cc:2468] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:28.737557 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c181adf9db574ab39432b2f9b14fe4d7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:28.812788 14607 raft_consensus.cc:3060] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.822768 14905 ts_tablet_manager.cc:1916] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:28.823712 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c181adf9db574ab39432b2f9b14fe4d7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:28.824280 14296 raft_consensus.cc:3060] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.824419 14703 tablet_bootstrap.cc:492] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:28.825009 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b60d99c2b4f442da4a4d9c5190f5797" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:28.825511 14299 raft_consensus.cc:2468] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:28.829329 14703 tablet_bootstrap.cc:654] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.829341 14904 tablet_replica.cc:333] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:28.830212 14904 raft_consensus.cc:2243] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:28.830693 14904 raft_consensus.cc:2272] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:28.833045 14904 ts_tablet_manager.cc:1916] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:28.839361 14385 leader_election.cc:304] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:28.846935 14722 tablet_bootstrap.cc:492] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:28.847596 14864 raft_consensus.cc:2804] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:28.847904 14864 raft_consensus.cc:493] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:28.848258 14864 raft_consensus.cc:3060] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.855376 14850 raft_consensus.cc:515] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.859506 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9524aa1503549bd820918d1c025aa2e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:28.860034 14449 raft_consensus.cc:3060] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.875598 14893 raft_consensus.cc:493] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:28.876245 14893 raft_consensus.cc:515] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.879208 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9524aa1503549bd820918d1c025aa2e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:28.879899 14608 raft_consensus.cc:3060] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:28.883778 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38655ee4996e41cc8f87b5248264613d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:28.887619 14542 leader_election.cc:343] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:28.889817 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38655ee4996e41cc8f87b5248264613d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:28.893635 14544 leader_election.cc:343] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:28.894193 14544 leader_election.cc:304] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:28.895310 14881 raft_consensus.cc:2749] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:28.896260 14893 leader_election.cc:290] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:28.897369 14607 raft_consensus.cc:2468] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:28.899032 14382 leader_election.cc:304] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:28.900112 14862 raft_consensus.cc:2804] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:28.900645 14862 raft_consensus.cc:697] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:28.901623 14862 consensus_queue.cc:237] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.909482 14850 leader_election.cc:290] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:28.914295 14722 tablet_bootstrap.cc:654] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:28.916802 14866 consensus_queue.cc:1048] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:28.937088 14714 tablet_bootstrap.cc:492] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:28.937912 14714 ts_tablet_manager.cc:1403] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.560s user 0.010s sys 0.002s
I20251211 08:57:28.943024 14449 raft_consensus.cc:2468] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:28.945004 14864 raft_consensus.cc:515] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:28.946859 14864 leader_election.cc:290] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:28.947072 13999 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 -> Peer 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015): Couldn't send request to peer 73702d54989f4959acfc4a6fdcede6ca. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251211 08:57:28.947165 14296 raft_consensus.cc:2468] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
W20251211 08:57:28.947590 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.62:39015, user_credentials={real_user=slave}} blocked reactor thread for 124171us
I20251211 08:57:28.980702 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 53fc34c41afe44678b8d6dcf79a31f8b (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]). Expected version 2 got 0
I20251211 08:57:28.981374 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 1a2dcb9ac62e4aac885d638f4e2b072e (table default.Table_b6bdd942603d47ef84a718099c2e7c40 [id=1673af33c2f846c7bf092303eff6e8ad]). Expected version 1 got 0
I20251211 08:57:28.982473 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 593df63696a04f85b00c301d93748e80 (table default.Table_6b8cf2aad2794338bd00ec9e24500365 [id=ab93470ba8f54070be718c240d49aa13]). Expected version 1 got 0
I20251211 08:57:28.987891 14714 raft_consensus.cc:359] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.004925 14714 raft_consensus.cc:385] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:29.005287 14714 raft_consensus.cc:740] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:29.005908 14714 consensus_queue.cc:260] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.009584 14722 tablet_bootstrap.cc:492] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:29.009966 14722 ts_tablet_manager.cc:1403] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.163s user 0.006s sys 0.005s
I20251211 08:57:29.012040 14722 raft_consensus.cc:359] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.012609 14722 raft_consensus.cc:385] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:29.012844 14722 raft_consensus.cc:740] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:29.014483 14011 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 28c6aa772db04d6c90b9a102d65f5681
I20251211 08:57:29.015645 14714 ts_tablet_manager.cc:1434] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.073s user 0.006s sys 0.000s
I20251211 08:57:29.016855 14714 tablet_bootstrap.cc:492] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:29.023015 14722 consensus_queue.cc:260] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.033594 14722 ts_tablet_manager.cc:1434] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20251211 08:57:29.033195 14009 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_c76caa256ffd4898a64d730f6583004f" } modify_external_catalogs: true
I20251211 08:57:29.033931 14009 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_c76caa256ffd4898a64d730f6583004f" } modify_external_catalogs: true
I20251211 08:57:29.034312 14703 tablet_bootstrap.cc:492] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:29.034696 14703 ts_tablet_manager.cc:1403] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.210s user 0.058s sys 0.089s
I20251211 08:57:29.036720 14703 raft_consensus.cc:359] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.037259 14703 raft_consensus.cc:385] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:29.037488 14703 raft_consensus.cc:740] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:29.038094 14703 consensus_queue.cc:260] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.043111 14703 ts_tablet_manager.cc:1434] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20251211 08:57:29.043869 14703 tablet_bootstrap.cc:492] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:29.044437 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_eb447b641c624aeba08c478e373a532d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:29.046028 14608 raft_consensus.cc:2468] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
W20251211 08:57:29.046269 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb447b641c624aeba08c478e373a532d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:29.047385 14230 leader_election.cc:304] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:29.048106 14850 raft_consensus.cc:2804] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:29.048759 14703 tablet_bootstrap.cc:654] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:29.076365 14703 tablet_bootstrap.cc:492] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:29.076805 14703 ts_tablet_manager.cc:1403] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20251211 08:57:29.079977 14722 tablet_bootstrap.cc:492] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:29.086063 14722 tablet_bootstrap.cc:654] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:29.091356 14703 raft_consensus.cc:359] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.092008 14703 raft_consensus.cc:385] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:29.092262 14703 raft_consensus.cc:740] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:29.089257 14850 raft_consensus.cc:697] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:29.093154 14850 consensus_queue.cc:237] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.099553 14703 consensus_queue.cc:260] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.112812 14703 ts_tablet_manager.cc:1434] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20251211 08:57:29.113744 14703 tablet_bootstrap.cc:492] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:29.115360 14893 raft_consensus.cc:493] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:29.115788 14893 raft_consensus.cc:515] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.117405 14893 leader_election.cc:290] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:29.120020 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab1ee5cf4eb4e11a72f5ad1de8c3856" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:29.122161 14544 leader_election.cc:343] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:29.129390 14703 tablet_bootstrap.cc:654] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:29.133005 14006 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 426e26859b9c4966b8ec8ba3eab53a43 (table default.Table_a144e458c8554442a03170f402a13d01 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 1 got 0
I20251211 08:57:29.133705 14006 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet a0f9c321e673433498a9059dd2e2efe6 (table default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5]). Expected version 1 got 0
I20251211 08:57:29.142971 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 1853070f81c34b52b8c74a7402e6dad3 (table default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4]). Expected version 1 got 0
I20251211 08:57:29.143651 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet a0f9c321e673433498a9059dd2e2efe6 (table default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5]). Expected version 1 got 0
I20251211 08:57:29.147289 14904 ts_tablet_manager.cc:1929] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:29.147670 14904 log.cc:1199] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/d582d092ce05451e9ef541c56de6e387
I20251211 08:57:29.148427 14904 ts_tablet_manager.cc:1950] T d582d092ce05451e9ef541c56de6e387 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:29.149238 14905 ts_tablet_manager.cc:1929] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:29.149564 14905 log.cc:1199] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/d582d092ce05451e9ef541c56de6e387
I20251211 08:57:29.149682 14012 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55614 (request call id 88) took 1403 ms. Trace:
I20251211 08:57:29.150213 14905 ts_tablet_manager.cc:1950] T d582d092ce05451e9ef541c56de6e387 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:57:29.157573 14010 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 21) took 1349 ms (client timeout 1000 ms). Trace:
W20251211 08:57:29.161321 14013 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 29) took 1323 ms (client timeout 1000 ms). Trace:
W20251211 08:57:29.161610 14013 rpcz_store.cc:269] 1211 08:57:27.837451 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:27.837583 (+ 132us) service_pool.cc:225] Handling call
1211 08:57:29.161255 (+1323672us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19,"reported_tablets":5}
I20251211 08:57:29.164354 14703 tablet_bootstrap.cc:492] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:29.164745 14703 ts_tablet_manager.cc:1403] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.002s
I20251211 08:57:29.166983 14703 raft_consensus.cc:359] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.167749 14703 raft_consensus.cc:385] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:29.150022 14012 rpcz_store.cc:276] 1211 08:57:27.746073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:27.746210 (+ 137us) service_pool.cc:225] Handling call
1211 08:57:27.783511 (+ 37301us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da87016 00007fc20b6b5b24 00007fc20b6b5ab2 00007fc20b6b59e1 00007fc20b6b595c 00007fc20b6b4c12 00007fc20998455c 00007fc209983996
1211 08:57:27.784252 (+ 741us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:27.784920 (+ 668us) catalog_manager.cc:3708] Apply alter schema
1211 08:57:27.785566 (+ 646us) catalog_manager.cc:3740] Acquired catalog manager lock
1211 08:57:27.785735 (+ 169us) catalog_manager.cc:3898] Updating metadata on disk
1211 08:57:28.038462 (+252727us) catalog_manager.cc:3929] Committing alterations to in-memory state
1211 08:57:29.149517 (+1111055us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:27.880159 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:27.880519 (+ 360us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:27.880542 (+ 23us) write_op.cc:435] Acquired schema lock
1211 08:57:27.880555 (+ 13us) tablet.cc:660] Decoding operations
1211 08:57:27.880828 (+ 273us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:27.880861 (+ 33us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:27.880886 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:27.881007 (+ 121us) tablet.cc:699] Row locks acquired
1211 08:57:27.881020 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:27.881110 (+ 90us) write_op.cc:270] Start()
1211 08:57:27.881187 (+ 77us) write_op.cc:276] Timestamp: P: 1765443447881031 usec, L: 0
1211 08:57:27.881202 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:57:27.881594 (+ 392us) log.cc:844] Serialized 404 byte log entry
1211 08:57:28.036947 (+155353us) op_driver.cc:464] REPLICATION: finished
1211 08:57:28.037161 (+ 214us) write_op.cc:301] APPLY: starting
1211 08:57:28.037210 (+ 49us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:28.037408 (+ 198us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:28.037492 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:28.037749 (+ 257us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:28.037868 (+ 119us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:28.037910 (+ 42us) write_op.cc:312] APPLY: finished
1211 08:57:28.038061 (+ 151us) log.cc:844] Serialized 22 byte log entry
1211 08:57:28.038181 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:28.038241 (+ 60us) write_op.cc:454] Released schema lock
1211 08:57:28.038276 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":83983232,"child_traces":[["op",{"apply.queue_time_us":74,"apply.run_cpu_time_us":1101,"apply.run_wall_time_us":1746,"num_ops":1,"prepare.queue_time_us":93176,"prepare.run_cpu_time_us":1601,"prepare.run_wall_time_us":1600,"replication_time_us":155660}]]}
I20251211 08:57:29.168040 14703 raft_consensus.cc:740] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
W20251211 08:57:29.169418 14007 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 18) took 2182 ms (client timeout 1000 ms). Trace:
W20251211 08:57:29.169759 14007 rpcz_store.cc:269] 1211 08:57:26.972162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:26.975990 (+ 3828us) service_pool.cc:225] Handling call
1211 08:57:29.154703 (+2178713us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:27.788247 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:27.788572 (+ 325us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:27.788589 (+ 17us) write_op.cc:435] Acquired schema lock
1211 08:57:27.788599 (+ 10us) tablet.cc:660] Decoding operations
1211 08:57:27.788846 (+ 247us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:27.788941 (+ 95us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:27.788962 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:27.789500 (+ 538us) tablet.cc:699] Row locks acquired
1211 08:57:27.789512 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:57:27.790653 (+ 1141us) write_op.cc:270] Start()
1211 08:57:27.790752 (+ 99us) write_op.cc:276] Timestamp: P: 1765443447790575 usec, L: 0
1211 08:57:27.790768 (+ 16us) op_driver.cc:348] REPLICATION: starting
1211 08:57:27.791229 (+ 461us) log.cc:844] Serialized 906 byte log entry
1211 08:57:27.882097 (+ 90868us) op_driver.cc:464] REPLICATION: finished
1211 08:57:27.882398 (+ 301us) write_op.cc:301] APPLY: starting
1211 08:57:27.882448 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:27.882568 (+ 120us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:27.882670 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:27.882959 (+ 289us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:27.883104 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:27.883155 (+ 51us) write_op.cc:312] APPLY: finished
1211 08:57:27.883335 (+ 180us) log.cc:844] Serialized 28 byte log entry
1211 08:57:27.883486 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:27.883551 (+ 65us) write_op.cc:454] Released schema lock
1211 08:57:27.883583 (+ 32us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:27.883809 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:27.883834 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":156,"apply.run_cpu_time_us":1376,"apply.run_wall_time_us":1376,"num_ops":2,"prepare.queue_time_us":221851,"prepare.run_cpu_time_us":2112,"prepare.run_wall_time_us":30684,"raft.queue_time_us":761,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":91227,"thread_start_us":26371,"threads_started":1}]]}
I20251211 08:57:29.174328 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_8dd853c2cd9f4a18816d6724773e96cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:29.175017 14703 consensus_queue.cc:260] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:29.175904 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dd853c2cd9f4a18816d6724773e96cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:29.178560 14722 tablet_bootstrap.cc:492] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:29.178975 14722 ts_tablet_manager.cc:1403] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.099s user 0.015s sys 0.009s
W20251211 08:57:29.157940 14010 rpcz_store.cc:269] 1211 08:57:27.808475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:27.808622 (+ 147us) service_pool.cc:225] Handling call
1211 08:57:27.833350 (+ 24728us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:29.157387 (+1324037us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:28.801460 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:28.801794 (+ 334us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:28.801819 (+ 25us) write_op.cc:435] Acquired schema lock
1211 08:57:28.801835 (+ 16us) tablet.cc:660] Decoding operations
1211 08:57:28.802037 (+ 202us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:28.802068 (+ 31us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:28.802088 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:28.802202 (+ 114us) tablet.cc:699] Row locks acquired
1211 08:57:28.802214 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:57:28.802306 (+ 92us) write_op.cc:270] Start()
1211 08:57:28.802373 (+ 67us) write_op.cc:276] Timestamp: P: 1765443448802234 usec, L: 0
1211 08:57:28.802386 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:28.802739 (+ 353us) log.cc:844] Serialized 529 byte log entry
1211 08:57:28.966643 (+163904us) op_driver.cc:464] REPLICATION: finished
1211 08:57:28.973789 (+ 7146us) write_op.cc:301] APPLY: starting
1211 08:57:28.973841 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:28.973930 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:28.974014 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:28.974155 (+ 141us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:28.974284 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:28.974409 (+ 125us) write_op.cc:312] APPLY: finished
1211 08:57:28.974552 (+ 143us) log.cc:844] Serialized 22 byte log entry
1211 08:57:28.974667 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:28.974743 (+ 76us) write_op.cc:454] Released schema lock
1211 08:57:28.974772 (+ 29us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:28.998337 (+ 23565us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:28.998363 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":52337920,"child_traces":[["op",{"apply.queue_time_us":7024,"apply.run_cpu_time_us":1206,"apply.run_wall_time_us":24416,"mutex_wait_us":126922,"num_ops":1,"prepare.queue_time_us":130320,"prepare.run_cpu_time_us":1473,"prepare.run_wall_time_us":1551,"replication_time_us":164168}]]}
I20251211 08:57:29.181993 14722 raft_consensus.cc:359] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.182560 14722 raft_consensus.cc:385] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:29.182803 14722 raft_consensus.cc:740] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:29.183390 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 7d4f6727c4464cc9b7d68866c3969a0f
I20251211 08:57:29.183590 14722 consensus_queue.cc:260] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.187654 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d4f6727c4464cc9b7d68866c3969a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:29.189003 14703 ts_tablet_manager.cc:1434] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20251211 08:57:29.193337 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.62:39015
--webserver_interface=127.13.146.62
--webserver_port=43581
--builtin_ntp_servers=127.13.146.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251211 08:57:29.203042 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d4f6727c4464cc9b7d68866c3969a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:29.205793 14005 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 19) took 985 ms (client timeout 1000 ms). Trace:
W20251211 08:57:29.206204 14005 rpcz_store.cc:269] 1211 08:57:28.220373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:28.220510 (+ 137us) service_pool.cc:225] Handling call
1211 08:57:29.205612 (+985102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:28.568237 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:28.568623 (+ 386us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:28.568646 (+ 23us) write_op.cc:435] Acquired schema lock
1211 08:57:28.568658 (+ 12us) tablet.cc:660] Decoding operations
1211 08:57:28.568897 (+ 239us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:28.568935 (+ 38us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:28.568969 (+ 34us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:28.569094 (+ 125us) tablet.cc:699] Row locks acquired
1211 08:57:28.569106 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:57:28.569198 (+ 92us) write_op.cc:270] Start()
1211 08:57:28.569269 (+ 71us) write_op.cc:276] Timestamp: P: 1765443448569145 usec, L: 0
1211 08:57:28.569283 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:28.569727 (+ 444us) log.cc:844] Serialized 529 byte log entry
1211 08:57:28.659867 (+ 90140us) op_driver.cc:464] REPLICATION: finished
1211 08:57:28.660678 (+ 811us) write_op.cc:301] APPLY: starting
1211 08:57:28.660732 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:28.660820 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:28.660942 (+ 122us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:28.661079 (+ 137us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:28.661230 (+ 151us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:28.661269 (+ 39us) write_op.cc:312] APPLY: finished
1211 08:57:28.661463 (+ 194us) log.cc:844] Serialized 22 byte log entry
1211 08:57:28.662049 (+ 586us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:28.662114 (+ 65us) write_op.cc:454] Released schema lock
1211 08:57:28.662144 (+ 30us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:28.795551 (+133407us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:28.795576 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":648,"apply.run_cpu_time_us":1282,"apply.run_wall_time_us":134838,"num_ops":1,"prepare.queue_time_us":2496,"prepare.run_cpu_time_us":1752,"prepare.run_wall_time_us":18450,"raft.queue_time_us":330,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":12,"replication_time_us":90498}]]}
I20251211 08:57:29.211270 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b997f016aa90456c8c6b1a4dcf5f4c75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:29.219511 14703 tablet_bootstrap.cc:492] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
W20251211 08:57:29.221706 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b997f016aa90456c8c6b1a4dcf5f4c75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:29.222406 14893 raft_consensus.cc:493] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:29.222836 14893 raft_consensus.cc:515] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.224526 14893 leader_election.cc:290] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:29.224748 14881 raft_consensus.cc:493] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:29.225148 14881 raft_consensus.cc:515] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.226811 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2c6080b3c1e4897bc1881e50f29f3a0" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:29.226884 14881 leader_election.cc:290] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:29.227798 14544 leader_election.cc:343] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:29.228901 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559f9d428b5841708999725b5b859660" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:29.230545 14544 leader_election.cc:343] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:29.232520 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 15f7bcd680064b9f8fe51aa05acce6fb (DEFAULT_TABLE table=default.Table_d1d406b278304b5ca6e6cc3b63e7fd36 [id=4208ab42111742f38028caeb7913a1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.233457 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f7bcd680064b9f8fe51aa05acce6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.233629 14703 tablet_bootstrap.cc:654] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:29.238508 14722 ts_tablet_manager.cc:1434] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.059s user 0.004s sys 0.000s
I20251211 08:57:29.248878 14904 tablet_replica.cc:333] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:29.249639 14904 raft_consensus.cc:2243] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:29.250077 14904 raft_consensus.cc:2272] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:29.252784 14905 tablet_replica.cc:333] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:29.253062 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet d582d092ce05451e9ef541c56de6e387 (table default.Table_ea8ceaf3dcc7478ea8aceec86870f8dd [id=deaf44db4c1146a8ac640aa3ef778467]) successfully deleted
I20251211 08:57:29.253608 14905 raft_consensus.cc:2243] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:29.252434 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_afa03065224d44a3a321c94b59f12587"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:29.254041 14905 raft_consensus.cc:2272] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:29.254966 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet d582d092ce05451e9ef541c56de6e387 (table default.Table_ea8ceaf3dcc7478ea8aceec86870f8dd [id=deaf44db4c1146a8ac640aa3ef778467]) successfully deleted
W20251211 08:57:29.255578 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet d235bade012540e1bb1c0dd871eff1bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d235bade012540e1bb1c0dd871eff1bb already in progress: creating tablet
W20251211 08:57:29.256430 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afa03065224d44a3a321c94b59f12587 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:29.256490 14905 ts_tablet_manager.cc:1916] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:29.260205 14904 ts_tablet_manager.cc:1916] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:29.224323 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_86772d5552714163a6a0ffc236521623" } modify_external_catalogs: true
I20251211 08:57:29.263315 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_86772d5552714163a6a0ffc236521623" } modify_external_catalogs: true
I20251211 08:57:29.269699 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 15f7bcd680064b9f8fe51aa05acce6fb (DEFAULT_TABLE table=default.Table_d1d406b278304b5ca6e6cc3b63e7fd36 [id=4208ab42111742f38028caeb7913a1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.270658 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f7bcd680064b9f8fe51aa05acce6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.271467 14935 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:29.279521 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_04406fa17cb54541b721e2b11e4b831a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:29.280962 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04406fa17cb54541b721e2b11e4b831a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:29.307755 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 996c85729acd4418be1763b73cbf7d2d (DEFAULT_TABLE table=default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.319562 14939 tablet.cc:1722] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:29.325933 14938 raft_consensus.cc:493] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:29.328780 14934 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:29.332798 14938 raft_consensus.cc:515] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.335032 14938 leader_election.cc:290] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:29.356637 14012 master_service.cc:954] ReplaceTablet: received request to replace tablet 53fc34c41afe44678b8d6dcf79a31f8b from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:29.363446 14010 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_47c60a996b0642519864b928857b2c86" } new_table_name: "default.Table_a939f5e4faf546148710852f96ded5d8" modify_external_catalogs: true
I20251211 08:57:29.367034 14004 master_service.cc:954] ReplaceTablet: received request to replace tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:29.370996 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_582685f00d3f4077bb184ec676e2ad02" } new_table_name: "default.Table_9e37e9a3471a486692ffe27b9841b13a" modify_external_catalogs: true
I20251211 08:57:29.370220 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 996c85729acd4418be1763b73cbf7d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.399406 14722 tablet_bootstrap.cc:492] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:29.402155 14703 tablet_bootstrap.cc:492] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:29.402544 14703 ts_tablet_manager.cc:1403] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.183s user 0.009s sys 0.016s
I20251211 08:57:29.404598 14703 raft_consensus.cc:359] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.405143 14703 raft_consensus.cc:385] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:29.405350 14703 raft_consensus.cc:740] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:29.405815 14703 consensus_queue.cc:260] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.408201 14009 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f647a2c75e7c46939a62573dea122fef" } modify_external_catalogs: true
I20251211 08:57:29.408852 14009 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f647a2c75e7c46939a62573dea122fef" } modify_external_catalogs: true
W20251211 08:57:29.419904 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 121506us
I20251211 08:57:29.423693 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0f9c321e673433498a9059dd2e2efe6" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:29.424474 14608 raft_consensus.cc:2468] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
W20251211 08:57:29.426890 14382 leader_election.cc:343] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 7b60d99c2b4f442da4a4d9c5190f5797
I20251211 08:57:29.429158 14382 leader_election.cc:304] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:29.429984 14938 raft_consensus.cc:2804] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:29.430363 14938 raft_consensus.cc:493] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:29.430660 14938 raft_consensus.cc:3060] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:29.435433 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 996c85729acd4418be1763b73cbf7d2d (DEFAULT_TABLE table=default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.438439 14275 tablet_service.cc:1505] Processing CreateTablet for tablet e354416047f74d02a9455147ca8d6bf3 (DEFAULT_TABLE table=default.Table_2670ab0571da4c8f8ccbeee370ba3914 [id=491e5bc8d1524a7aae54a99a4d39c7f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.439580 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e354416047f74d02a9455147ca8d6bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.440403 14703 ts_tablet_manager.cc:1434] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.038s user 0.003s sys 0.001s
I20251211 08:57:29.441174 14703 tablet_bootstrap.cc:492] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:29.454594 14938 raft_consensus.cc:515] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.456525 14938 leader_election.cc:290] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:29.457002 14938 consensus_queue.cc:1048] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:29.461305 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0f9c321e673433498a9059dd2e2efe6" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:29.461753 14607 raft_consensus.cc:3060] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:29.486147 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0f9c321e673433498a9059dd2e2efe6" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:29.486721 14298 raft_consensus.cc:2468] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:29.490402 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b60d99c2b4f442da4a4d9c5190f5797" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:29.491183 14296 raft_consensus.cc:3060] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:29.504787 14423 tablet_service.cc:1505] Processing CreateTablet for tablet 996c85729acd4418be1763b73cbf7d2d (DEFAULT_TABLE table=default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.506487 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 996c85729acd4418be1763b73cbf7d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.507043 14881 raft_consensus.cc:493] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:29.507558 14881 raft_consensus.cc:515] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.511240 14703 tablet_bootstrap.cc:654] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:29.516692 14881 leader_election.cc:290] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:29.517957 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5191443773b415198687c55c4517692" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:29.518524 14294 raft_consensus.cc:2468] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:29.520160 14544 leader_election.cc:304] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:29.521059 14881 raft_consensus.cc:2804] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:29.521426 14881 raft_consensus.cc:493] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:29.521792 14881 raft_consensus.cc:3060] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:29.543692 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet e5a41643bad84006819fec986657e4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:29.544353 14429 tablet_service.cc:1505] Processing CreateTablet for tablet 15f7bcd680064b9f8fe51aa05acce6fb (DEFAULT_TABLE table=default.Table_d1d406b278304b5ca6e6cc3b63e7fd36 [id=4208ab42111742f38028caeb7913a1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.544646 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet d582d092ce05451e9ef541c56de6e387 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:29.544847 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f112603316047289459698ec28b54d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:26 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:29.543778 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d4f6727c4464cc9b7d68866c3969a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:29.545518 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f7bcd680064b9f8fe51aa05acce6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.557085 14443 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5191443773b415198687c55c4517692" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:29.559751 14542 leader_election.cc:343] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:29.560200 14607 raft_consensus.cc:2468] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:29.562085 14382 leader_election.cc:304] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:29.563197 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 996c85729acd4418be1763b73cbf7d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.567739 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559f9d428b5841708999725b5b859660" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:29.568565 14444 raft_consensus.cc:2468] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:29.569808 14542 leader_election.cc:304] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:29.570823 14893 raft_consensus.cc:2804] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:29.571198 14893 raft_consensus.cc:493] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:29.571509 14893 raft_consensus.cc:3060] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:29.572820 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2c6080b3c1e4897bc1881e50f29f3a0" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:29.574512 14542 leader_election.cc:343] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:29.575002 14542 leader_election.cc:304] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:29.590193 14945 raft_consensus.cc:2749] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:29.591738 14922 raft_consensus.cc:493] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:29.591979 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab1ee5cf4eb4e11a72f5ad1de8c3856" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:29.592229 14922 raft_consensus.cc:515] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:29.598192 14542 leader_election.cc:343] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:29.598766 14542 leader_election.cc:304] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:29.608618 14718 tablet_replica.cc:333] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:29.609287 14864 raft_consensus.cc:2804] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:29.609759 14864 raft_consensus.cc:697] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:29.614418 14945 raft_consensus.cc:2749] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:29.623970 14586 tablet_service.cc:1505] Processing CreateTablet for tablet e354416047f74d02a9455147ca8d6bf3 (DEFAULT_TABLE table=default.Table_2670ab0571da4c8f8ccbeee370ba3914 [id=491e5bc8d1524a7aae54a99a4d39c7f2]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:29.625260 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet e5a41643bad84006819fec986657e4ab because the tablet was not found. No further retry: Not found: Tablet not found: e5a41643bad84006819fec986657e4ab
W20251211 08:57:29.650593 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet d582d092ce05451e9ef541c56de6e387 because the tablet was not found. No further retry: Not found: Tablet not found: d582d092ce05451e9ef541c56de6e387
I20251211 08:57:29.652999 14714 tablet_bootstrap.cc:700] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: Time spent opening tablet: real 0.626s user 0.004s sys 0.000s
I20251211 08:57:29.654060 14714 tablet_bootstrap.cc:654] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:29.655753 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 8f112603316047289459698ec28b54d4 because the tablet was not found. No further retry: Not found: Tablet not found: 8f112603316047289459698ec28b54d4
I20251211 08:57:29.610785 14864 consensus_queue.cc:237] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.661350 14905 ts_tablet_manager.cc:1929] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:29.661701 14905 log.cc:1199] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/d235bade012540e1bb1c0dd871eff1bb
I20251211 08:57:29.662283 14905 ts_tablet_manager.cc:1950] T d235bade012540e1bb1c0dd871eff1bb P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:29.663980 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e354416047f74d02a9455147ca8d6bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.681370 14945 raft_consensus.cc:493] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:29.681830 14945 raft_consensus.cc:515] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.683981 14881 raft_consensus.cc:515] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.685277 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00f79094ed8b4782a3a1488e41794f9b" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:29.685817 14881 leader_election.cc:290] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:29.687067 14544 leader_election.cc:343] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:29.688289 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5191443773b415198687c55c4517692" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:29.688831 14294 raft_consensus.cc:3060] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:29.700569 14866 consensus_queue.cc:1048] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:29.704540 14945 leader_election.cc:290] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:29.720755 14011 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a] alter complete (version 2)
I20251211 08:57:29.722390 14428 tablet_service.cc:1505] Processing CreateTablet for tablet e354416047f74d02a9455147ca8d6bf3 (DEFAULT_TABLE table=default.Table_2670ab0571da4c8f8ccbeee370ba3914 [id=491e5bc8d1524a7aae54a99a4d39c7f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.723456 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e354416047f74d02a9455147ca8d6bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.725417 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5191443773b415198687c55c4517692" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:29.726974 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00f79094ed8b4782a3a1488e41794f9b" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:29.731308 14940 tablet.cc:1722] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:29.732271 14722 tablet_bootstrap.cc:654] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:29.733870 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0f9c321e673433498a9059dd2e2efe6" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:29.734529 14298 raft_consensus.cc:3060] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:29.735823 14922 leader_election.cc:290] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:29.736403 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45f3395a34cf4fdd8e1898b08d123c8a" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:29.736963 14607 raft_consensus.cc:2468] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:29.738231 14230 leader_election.cc:304] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:29.738611 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 38655ee4996e41cc8f87b5248264613d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:29.740520 14850 raft_consensus.cc:2804] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:29.740834 14850 raft_consensus.cc:493] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:29.741174 14850 raft_consensus.cc:3060] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:29.741642 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45f3395a34cf4fdd8e1898b08d123c8a" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:29.742259 14542 leader_election.cc:343] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:29.742646 14231 leader_election.cc:343] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:29.741658 14385 consensus_peers.cc:597] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:29.742914 14542 leader_election.cc:343] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:29.744966 14862 consensus_queue.cc:1048] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:29.745553 14922 raft_consensus.cc:493] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:29.745801 14714 tablet_bootstrap.cc:492] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:29.746273 14714 ts_tablet_manager.cc:1403] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.730s user 0.009s sys 0.003s
I20251211 08:57:29.746455 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet c5191443773b415198687c55c4517692 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:29.748535 14714 raft_consensus.cc:359] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.743270 14542 leader_election.cc:304] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:29.749063 14714 raft_consensus.cc:385] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:29.746047 14922 raft_consensus.cc:515] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.749322 14714 raft_consensus.cc:740] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
W20251211 08:57:29.749992 14006 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 22) took 929 ms (client timeout 1000 ms). Trace:
I20251211 08:57:29.749835 14714 consensus_queue.cc:260] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:29.750348 14006 rpcz_store.cc:269] 1211 08:57:28.820253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:28.830903 (+ 10650us) service_pool.cc:225] Handling call
1211 08:57:29.749923 (+919020us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":14}
I20251211 08:57:29.751624 14714 ts_tablet_manager.cc:1434] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251211 08:57:29.753391 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "426e26859b9c4966b8ec8ba3eab53a43" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:29.753926 14447 raft_consensus.cc:2468] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:29.754840 14231 leader_election.cc:304] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:29.757689 14718 raft_consensus.cc:2243] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:29.758571 14718 raft_consensus.cc:2272] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:29.765347 14893 raft_consensus.cc:515] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.766300 14881 raft_consensus.cc:2749] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:29.767518 14893 leader_election.cc:290] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:29.771862 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 60) took 1193 ms. Trace:
I20251211 08:57:29.772207 14905 rpcz_store.cc:276] 1211 08:57:28.578060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:28.578175 (+ 115us) service_pool.cc:225] Handling call
1211 08:57:29.252635 (+674460us) ts_tablet_manager.cc:1151] Deleting tablet d235bade012540e1bb1c0dd871eff1bb
1211 08:57:29.252670 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:29.458873 (+206203us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:29.661099 (+202226us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:29.771807 (+110708us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5795,"tablet-delete.queue_time_us":673898}
I20251211 08:57:29.773728 14905 tablet_replica.cc:333] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:29.774529 14905 raft_consensus.cc:2243] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:29.775368 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 60b23163906447c3877db631968f5623 (DEFAULT_TABLE table=default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.776276 14905 raft_consensus.cc:2272] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:29.778883 14905 ts_tablet_manager.cc:1916] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:29.784868 14298 raft_consensus.cc:2468] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:29.785993 14904 ts_tablet_manager.cc:1929] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:29.786439 14904 log.cc:1199] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/7d4f6727c4464cc9b7d68866c3969a0f
I20251211 08:57:29.787217 14904 ts_tablet_manager.cc:1950] T 7d4f6727c4464cc9b7d68866c3969a0f P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:29.791661 14922 leader_election.cc:290] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:29.794998 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559f9d428b5841708999725b5b859660" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:29.795879 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "426e26859b9c4966b8ec8ba3eab53a43" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:29.796447 14544 leader_election.cc:343] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:29.796658 14608 raft_consensus.cc:2468] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
W20251211 08:57:29.797394 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 54539.3us
I20251211 08:57:29.804243 14959 raft_consensus.cc:2804] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:29.804728 14959 raft_consensus.cc:493] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:29.805133 14959 raft_consensus.cc:3060] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:29.816584 14722 tablet_bootstrap.cc:492] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
W20251211 08:57:29.769820 13999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20251211 08:57:29.775216 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet d235bade012540e1bb1c0dd871eff1bb (table default.Table_7c70be2e41de4127a511a479fdb6ad7e [id=5b6d7c4fd998498f91fe1594382caed9]) successfully deleted
W20251211 08:57:29.819612 13997 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 44433us
I20251211 08:57:29.817095 14722 ts_tablet_manager.cc:1403] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.418s user 0.020s sys 0.019s
I20251211 08:57:29.824177 14722 raft_consensus.cc:359] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.824813 14722 raft_consensus.cc:385] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:29.825074 14722 raft_consensus.cc:740] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:29.825624 14722 consensus_queue.cc:260] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.828960 14714 tablet_bootstrap.cc:492] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:29.830062 14957 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251211 08:57:29.837980 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.62:39015, user_credentials={real_user=slave}} blocked reactor thread for 68209.3us
I20251211 08:57:29.841423 14703 tablet_bootstrap.cc:492] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:29.841885 14703 ts_tablet_manager.cc:1403] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.401s user 0.014s sys 0.041s
I20251211 08:57:29.845032 14722 ts_tablet_manager.cc:1434] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20251211 08:57:29.846817 14722 tablet_bootstrap.cc:492] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:29.851053 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 60b23163906447c3877db631968f5623 (DEFAULT_TABLE table=default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.851819 14722 tablet_bootstrap.cc:654] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:29.858201 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60b23163906447c3877db631968f5623. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.861433 14850 raft_consensus.cc:515] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.862708 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 25805a95349d49f58280a30e6876929d (DEFAULT_TABLE table=default.Table_94b52b113a9f45b69bff699f4d2bdd0f [id=52ae4637827d424eb4d37362bf21cea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.863662 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25805a95349d49f58280a30e6876929d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:29.865255 14277 consensus_meta.cc:287] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b: Time spent flushing consensus metadata: real 0.625s user 0.008s sys 0.000s
I20251211 08:57:29.866412 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60b23163906447c3877db631968f5623. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.868876 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b60d99c2b4f442da4a4d9c5190f5797 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:29.872318 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ac11e9583ef64c1bbae0b2940213b819"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:29.873946 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac11e9583ef64c1bbae0b2940213b819 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:29.875617 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 25805a95349d49f58280a30e6876929d (DEFAULT_TABLE table=default.Table_94b52b113a9f45b69bff699f4d2bdd0f [id=52ae4637827d424eb4d37362bf21cea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.868422 14703 raft_consensus.cc:359] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.879324 14703 raft_consensus.cc:385] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:29.879580 14703 raft_consensus.cc:740] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:29.881162 14703 consensus_queue.cc:260] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.882419 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6cbe43becdc846bcab4b1233f65cda0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:29.882851 14850 leader_election.cc:290] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:29.884618 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45f3395a34cf4fdd8e1898b08d123c8a" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:29.885063 14608 raft_consensus.cc:3060] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:29.885214 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_8dcae63f2e7547d290798fc98942d775"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:29.890189 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dcae63f2e7547d290798fc98942d775 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:29.886374 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2a5d1e8d612c4fc5954ea3efc00eb00b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:29.891882 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a5d1e8d612c4fc5954ea3efc00eb00b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:29.896364 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cbe43becdc846bcab4b1233f65cda0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:29.896255 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 60b23163906447c3877db631968f5623 (DEFAULT_TABLE table=default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:29.896587 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 103685us
I20251211 08:57:29.921655 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559f9d428b5841708999725b5b859660" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:29.922222 14449 raft_consensus.cc:3060] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:29.925784 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60b23163906447c3877db631968f5623. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:29.929374 14703 ts_tablet_manager.cc:1434] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.087s user 0.005s sys 0.001s
I20251211 08:57:29.928993 14938 consensus_queue.cc:1048] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20251211 08:57:29.934234 14941 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40738: Illegal state: RaftConsensus is not running: State = Stopped
I20251211 08:57:29.937335 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45f3395a34cf4fdd8e1898b08d123c8a" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:29.938314 14938 consensus_queue.cc:1048] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251211 08:57:29.940323 14850 consensus_queue.cc:1048] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:29.951795 14231 leader_election.cc:343] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:29.959107 14294 raft_consensus.cc:2468] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:29.966150 14904 tablet_replica.cc:333] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:29.954957 14296 raft_consensus.cc:2468] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:29.972599 14904 raft_consensus.cc:2243] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:29.973245 14904 raft_consensus.cc:2272] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:29.975412 14904 ts_tablet_manager.cc:1916] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:29.975577 14718 ts_tablet_manager.cc:1916] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:29.979336 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 7d4f6727c4464cc9b7d68866c3969a0f (table default.Table_c76caa256ffd4898a64d730f6583004f [id=e6bc81a67ba949c69798c5049da505b5]) successfully deleted
I20251211 08:57:29.979836 14967 tablet.cc:1722] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:29.988030 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 25805a95349d49f58280a30e6876929d (DEFAULT_TABLE table=default.Table_94b52b113a9f45b69bff699f4d2bdd0f [id=52ae4637827d424eb4d37362bf21cea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:29.954259 14608 raft_consensus.cc:2468] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:29.961421 14905 ts_tablet_manager.cc:1929] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:29.994829 14905 log.cc:1199] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/7d4f6727c4464cc9b7d68866c3969a0f
I20251211 08:57:29.995746 14703 tablet_bootstrap.cc:492] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:30.009296 14722 tablet_bootstrap.cc:492] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:30.009689 14722 ts_tablet_manager.cc:1403] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.164s user 0.008s sys 0.012s
I20251211 08:57:30.011806 14722 raft_consensus.cc:359] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.012372 14722 raft_consensus.cc:385] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.012621 14722 raft_consensus.cc:740] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.013491 14722 consensus_queue.cc:260] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:29.962311 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25805a95349d49f58280a30e6876929d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:30.016582 14722 ts_tablet_manager.cc:1434] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251211 08:57:30.017442 14722 tablet_bootstrap.cc:492] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:30.022156 14010 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5cf1210733754f43885a7c0a350aa206 has no consensus state
I20251211 08:57:30.022466 14722 tablet_bootstrap.cc:654] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:29.996903 14905 ts_tablet_manager.cc:1950] T 7d4f6727c4464cc9b7d68866c3969a0f P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:29.966882 14544 leader_election.cc:304] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:30.025099 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 53fc34c41afe44678b8d6dcf79a31f8b
I20251211 08:57:30.025045 14881 raft_consensus.cc:2804] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
W20251211 08:57:30.025179 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 58431.6us
I20251211 08:57:30.025509 14881 raft_consensus.cc:697] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:30.026451 14012 catalog_manager.cc:6724] ReplaceTablet: tablet 53fc34c41afe44678b8d6dcf79a31f8b deleted and replaced by tablet 383716a1f1f847ccbea542eca04790c9
I20251211 08:57:30.026242 14881 consensus_queue.cc:237] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.028427 14960 raft_consensus.cc:493] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:30.028803 14960 raft_consensus.cc:515] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.029459 14230 leader_election.cc:304] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:30.030424 14960 leader_election.cc:290] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:30.031560 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00f79094ed8b4782a3a1488e41794f9b consensus state has no leader
I20251211 08:57:30.032946 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet fab1ee5cf4eb4e11a72f5ad1de8c3856
I20251211 08:57:30.034250 14004 catalog_manager.cc:6724] ReplaceTablet: tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 deleted and replaced by tablet e6c4d4c99b3b496cab125e13153349cb
I20251211 08:57:30.037104 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet e015d0fb9d6c4c80a510c05244b334e4
I20251211 08:57:30.040822 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 with delete_type TABLET_DATA_DELETED (replaced by tablet e6c4d4c99b3b496cab125e13153349cb) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:30.040922 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 53fc34c41afe44678b8d6dcf79a31f8b with delete_type TABLET_DATA_DELETED (replaced by tablet 383716a1f1f847ccbea542eca04790c9) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:30.041427 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet e015d0fb9d6c4c80a510c05244b334e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:30.047245 14850 raft_consensus.cc:2804] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:30.047766 14850 raft_consensus.cc:697] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:30.048640 14850 consensus_queue.cc:237] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.052515 14722 tablet_bootstrap.cc:492] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:30.052922 14722 ts_tablet_manager.cc:1403] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.000s
I20251211 08:57:30.053633 14959 raft_consensus.cc:515] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.055338 14959 leader_election.cc:290] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:30.058967 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "426e26859b9c4966b8ec8ba3eab53a43" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:30.059477 14607 raft_consensus.cc:3060] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:30.062322 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet d2c6080b3c1e4897bc1881e50f29f3a0
I20251211 08:57:30.062656 14722 raft_consensus.cc:359] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.063287 14722 raft_consensus.cc:385] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251211 08:57:30.066767 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 5cf1210733754f43885a7c0a350aa206 (table default.Table_a939f5e4faf546148710852f96ded5d8 [id=e9973d6c9b1341d1b96f04367e44e212], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5cf1210733754f43885a7c0a350aa206 has no consensus state
I20251211 08:57:30.069677 14905 tablet_replica.cc:333] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:30.069906 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet d2c6080b3c1e4897bc1881e50f29f3a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:30.070433 14905 raft_consensus.cc:2243] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:30.071403 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 7d4f6727c4464cc9b7d68866c3969a0f (table default.Table_c76caa256ffd4898a64d730f6583004f [id=e6bc81a67ba949c69798c5049da505b5]) successfully deleted
I20251211 08:57:30.071636 14722 raft_consensus.cc:740] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.072751 14722 consensus_queue.cc:260] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.074689 14905 raft_consensus.cc:2272] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:30.076299 14722 ts_tablet_manager.cc:1434] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20251211 08:57:30.076624 14905 ts_tablet_manager.cc:1916] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:30.077823 14961 raft_consensus.cc:493] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:30.078203 14722 tablet_bootstrap.cc:492] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:30.078300 14961 raft_consensus.cc:515] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.086064 14961 leader_election.cc:290] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:30.087759 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab1ee5cf4eb4e11a72f5ad1de8c3856" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:30.088456 14608 raft_consensus.cc:2468] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:29.990579 14385 leader_election.cc:304] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:30.089372 14722 tablet_bootstrap.cc:654] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.101464 14703 tablet_bootstrap.cc:654] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.089561 14230 leader_election.cc:304] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:30.104529 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e015d0fb9d6c4c80a510c05244b334e4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:30.110049 14961 raft_consensus.cc:2804] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:30.110826 14961 raft_consensus.cc:493] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:30.111214 14961 raft_consensus.cc:3060] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:30.115634 14959 raft_consensus.cc:493] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:30.116039 14959 raft_consensus.cc:515] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.117802 14959 leader_election.cc:290] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:30.098591 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25805a95349d49f58280a30e6876929d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:30.120850 14279 tablet_service.cc:1552] Processing DeleteTablet for tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 with delete_type TABLET_DATA_DELETED (replaced by tablet e6c4d4c99b3b496cab125e13153349cb) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:30.121093 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet e015d0fb9d6c4c80a510c05244b334e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:30.121381 14274 tablet_service.cc:1552] Processing DeleteTablet for tablet 53fc34c41afe44678b8d6dcf79a31f8b with delete_type TABLET_DATA_DELETED (replaced by tablet 383716a1f1f847ccbea542eca04790c9) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:30.121469 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet d2c6080b3c1e4897bc1881e50f29f3a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:30.122354 14544 leader_election.cc:343] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.123737 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38655ee4996e41cc8f87b5248264613d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:30.125026 14230 leader_election.cc:343] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: Tablet not RUNNING: STOPPED
I20251211 08:57:30.123886 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e015d0fb9d6c4c80a510c05244b334e4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:30.127604 14542 leader_election.cc:343] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.127971 14542 leader_election.cc:304] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:30.128862 14960 raft_consensus.cc:2749] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:30.129609 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab1ee5cf4eb4e11a72f5ad1de8c3856" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:30.131954 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38655ee4996e41cc8f87b5248264613d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:30.133085 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "426e26859b9c4966b8ec8ba3eab53a43" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:30.133677 14446 raft_consensus.cc:3060] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:30.135099 14722 tablet_bootstrap.cc:492] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:30.135471 14722 ts_tablet_manager.cc:1403] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.057s user 0.014s sys 0.000s
I20251211 08:57:30.135459 14973 consensus_queue.cc:1048] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:30.136498 14862 raft_consensus.cc:2804] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:30.132074 14010 catalog_manager.cc:5654] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } }
I20251211 08:57:30.139663 14960 raft_consensus.cc:493] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:30.140040 14960 raft_consensus.cc:515] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:30.141407 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 150985us
I20251211 08:57:30.141742 14960 leader_election.cc:290] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:30.142231 14010 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 1853070f81c34b52b8c74a7402e6dad3 (table default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4]). Expected version 1 got 0
I20251211 08:57:30.142994 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 with delete_type TABLET_DATA_DELETED (replaced by tablet e6c4d4c99b3b496cab125e13153349cb) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:30.143021 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet e015d0fb9d6c4c80a510c05244b334e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:30.142638 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet 53fc34c41afe44678b8d6dcf79a31f8b with delete_type TABLET_DATA_DELETED (replaced by tablet 383716a1f1f847ccbea542eca04790c9) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:30.144790 14010 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 00f79094ed8b4782a3a1488e41794f9b (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]). Expected version 2 got 0
I20251211 08:57:30.145458 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b9b338a3f544ccb813247250b0f0eb7" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:30.145416 14010 catalog_manager.cc:5654] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } }
W20251211 08:57:30.146721 14542 leader_election.cc:343] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.147993 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b9b338a3f544ccb813247250b0f0eb7" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:30.148183 14423 consensus_meta.cc:287] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256: Time spent flushing consensus metadata: real 0.608s user 0.006s sys 0.002s
I20251211 08:57:30.099140 14449 raft_consensus.cc:2468] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
W20251211 08:57:30.149274 14544 leader_election.cc:343] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.149662 14544 leader_election.cc:304] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:30.151101 14607 raft_consensus.cc:2468] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:30.151798 14961 raft_consensus.cc:515] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.151955 14542 leader_election.cc:304] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:30.152643 14960 raft_consensus.cc:2749] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:30.153153 14960 raft_consensus.cc:2804] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:30.154145 14960 raft_consensus.cc:697] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:30.138693 14722 raft_consensus.cc:359] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.154529 14722 raft_consensus.cc:385] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.154886 14722 raft_consensus.cc:740] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.155035 14862 raft_consensus.cc:697] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:30.155007 14960 consensus_queue.cc:237] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.155853 14862 consensus_queue.cc:237] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.156281 14722 consensus_queue.cc:260] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.159575 14722 ts_tablet_manager.cc:1434] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20251211 08:57:30.159631 14961 leader_election.cc:290] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:30.160470 14722 tablet_bootstrap.cc:492] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:30.165042 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab1ee5cf4eb4e11a72f5ad1de8c3856" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:30.165057 14230 leader_election.cc:304] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:30.165617 14722 tablet_bootstrap.cc:654] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.165555 14607 raft_consensus.cc:3060] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:30.165942 14961 raft_consensus.cc:2804] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:30.166561 14961 raft_consensus.cc:697] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:30.167377 14961 consensus_queue.cc:237] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.173040 14703 tablet_bootstrap.cc:492] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:30.173521 14703 ts_tablet_manager.cc:1403] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.178s user 0.007s sys 0.003s
I20251211 08:57:30.176303 14703 raft_consensus.cc:359] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:30.195844 14231 leader_election.cc:343] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.204217 14422 tablet_service.cc:1552] Processing DeleteTablet for tablet d2c6080b3c1e4897bc1881e50f29f3a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:30.211081 14916 consensus_queue.cc:1048] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:30.216687 14703 raft_consensus.cc:385] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.217085 14703 raft_consensus.cc:740] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.223119 14703 consensus_queue.cc:260] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.225401 14703 ts_tablet_manager.cc:1434] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.052s user 0.005s sys 0.001s
I20251211 08:57:30.229842 14714 tablet_bootstrap.cc:654] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:30.240417 14231 leader_election.cc:343] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.240877 14231 leader_election.cc:304] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:30.242228 14922 raft_consensus.cc:2749] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:30.245447 14703 tablet_bootstrap.cc:492] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:30.258183 14904 ts_tablet_manager.cc:1929] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:30.258705 14904 log.cc:1199] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/7b60d99c2b4f442da4a4d9c5190f5797
I20251211 08:57:30.259541 14904 ts_tablet_manager.cc:1950] T 7b60d99c2b4f442da4a4d9c5190f5797 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:30.269670 14905 ts_tablet_manager.cc:1929] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:30.273985 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab1ee5cf4eb4e11a72f5ad1de8c3856" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:30.277469 14905 log.cc:1199] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/38655ee4996e41cc8f87b5248264613d
I20251211 08:57:30.278283 14905 ts_tablet_manager.cc:1950] T 38655ee4996e41cc8f87b5248264613d P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:57:30.279366 14231 leader_election.cc:343] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.285449 14703 tablet_bootstrap.cc:654] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.341256 14703 tablet_bootstrap.cc:492] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:30.341786 14703 ts_tablet_manager.cc:1403] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.096s user 0.010s sys 0.000s
I20251211 08:57:30.347728 14703 raft_consensus.cc:359] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.348549 14703 raft_consensus.cc:385] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.348863 14703 raft_consensus.cc:740] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.352917 14703 consensus_queue.cc:260] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.354846 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 7b60d99c2b4f442da4a4d9c5190f5797 (table default.Table_cbeef8d5f95540b9a51b10a0a18a2dce [id=71569e18b5d347598f45f7e38c073515]) successfully deleted
I20251211 08:57:30.355743 14607 raft_consensus.cc:2468] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:30.357227 14230 leader_election.cc:304] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:30.359233 14961 raft_consensus.cc:2804] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:30.360041 14961 raft_consensus.cc:697] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:30.360805 14961 consensus_queue.cc:237] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.362536 14905 tablet_replica.cc:333] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:30.363353 14905 raft_consensus.cc:2243] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:30.364041 14905 pending_rounds.cc:70] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:57:30.364324 14905 pending_rounds.cc:77] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256371330859008 op_type: NO_OP noop_request { }
I20251211 08:57:30.364691 14905 raft_consensus.cc:2889] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:30.365971 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 38655ee4996e41cc8f87b5248264613d (table default.Table_a7ade9d81bfc4498a43e0b5ad6941ac2 [id=c533026c25a54aa885eb483d4120f720]) successfully deleted
I20251211 08:57:30.365056 14905 raft_consensus.cc:2272] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:30.367861 14703 ts_tablet_manager.cc:1434] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.026s user 0.009s sys 0.000s
I20251211 08:57:30.369699 14905 ts_tablet_manager.cc:1916] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:30.354601 14904 tablet_replica.cc:333] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:30.375037 14960 raft_consensus.cc:493] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:30.375437 14960 raft_consensus.cc:515] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.377180 14960 leader_election.cc:290] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:30.377962 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_eb1d2cda272c4981a164a7db6301ec2c" } modify_external_catalogs: true
I20251211 08:57:30.378468 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_eb1d2cda272c4981a164a7db6301ec2c" } modify_external_catalogs: true
I20251211 08:57:30.381842 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12379255548f49dfad17f3840a0c86bb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:30.384214 14542 leader_election.cc:343] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.389717 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_1ac7df210845496091096027805f9fff" } new_table_name: "default.Table_9c85250afba741a79335663257915c82" modify_external_catalogs: true
I20251211 08:57:30.395489 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_20c0c6a80c5b467a917c0a9e8618440b" } new_table_name: "default.Table_cd86506c20bc42359393525a923146bd" modify_external_catalogs: true
I20251211 08:57:30.398929 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e478355b49e940ddaa167200d689b1e7" } modify_external_catalogs: true
I20251211 08:57:30.399386 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e478355b49e940ddaa167200d689b1e7" } modify_external_catalogs: true
I20251211 08:57:30.402283 14703 tablet_bootstrap.cc:492] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:30.410953 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12379255548f49dfad17f3840a0c86bb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:30.412549 14544 leader_election.cc:343] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.412981 14544 leader_election.cc:304] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:30.414636 14881 raft_consensus.cc:2749] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:30.433741 14703 tablet_bootstrap.cc:654] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.456226 14703 tablet_bootstrap.cc:492] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:30.456633 14703 ts_tablet_manager.cc:1403] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.000s
I20251211 08:57:30.458741 14703 raft_consensus.cc:359] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.459334 14703 raft_consensus.cc:385] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.459582 14703 raft_consensus.cc:740] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.467978 14011 master_service.cc:954] ReplaceTablet: received request to replace tablet e015d0fb9d6c4c80a510c05244b334e4 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:30.479068 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet 45f3395a34cf4fdd8e1898b08d123c8a from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:30.479434 14703 consensus_queue.cc:260] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.480347 14960 raft_consensus.cc:493] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:30.480743 14960 raft_consensus.cc:515] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.484508 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "996c85729acd4418be1763b73cbf7d2d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:30.485879 14544 leader_election.cc:343] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.487900 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "996c85729acd4418be1763b73cbf7d2d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:30.489075 14542 leader_election.cc:343] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.489595 14542 leader_election.cc:304] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:30.490537 14881 raft_consensus.cc:2749] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:30.491109 14960 leader_election.cc:290] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:30.496995 14714 tablet_bootstrap.cc:492] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:30.497623 14714 ts_tablet_manager.cc:1403] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.669s user 0.015s sys 0.000s
I20251211 08:57:30.503849 14703 ts_tablet_manager.cc:1434] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.047s user 0.005s sys 0.002s
I20251211 08:57:30.514006 14703 tablet_bootstrap.cc:492] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:30.531739 14703 tablet_bootstrap.cc:654] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.558100 14905 ts_tablet_manager.cc:1929] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:30.558487 14905 log.cc:1199] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/c5191443773b415198687c55c4517692
I20251211 08:57:30.559262 14905 ts_tablet_manager.cc:1950] T c5191443773b415198687c55c4517692 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:30.559437 14862 consensus_queue.cc:1048] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:30.560302 14718 ts_tablet_manager.cc:1929] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:30.561259 14718 log.cc:1199] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/7d4f6727c4464cc9b7d68866c3969a0f
I20251211 08:57:30.562150 14718 ts_tablet_manager.cc:1950] T 7d4f6727c4464cc9b7d68866c3969a0f P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:30.560868 14446 raft_consensus.cc:2468] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:30.617118 14938 consensus_queue.cc:1048] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:30.644428 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_81eee4b23e9045848867df8a0f511ae5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:30.646023 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81eee4b23e9045848867df8a0f511ae5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:30.534632 14714 raft_consensus.cc:359] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.651067 14714 raft_consensus.cc:385] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.651363 14714 raft_consensus.cc:740] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.651983 14714 consensus_queue.cc:260] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.653978 14714 ts_tablet_manager.cc:1434] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.156s user 0.004s sys 0.000s
I20251211 08:57:30.659298 14714 tablet_bootstrap.cc:492] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:30.660667 14905 tablet_replica.cc:333] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:30.661482 14905 raft_consensus.cc:2243] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:30.661888 14905 raft_consensus.cc:2272] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:30.664371 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet c5191443773b415198687c55c4517692 (table default.Table_4b521cb9e71f4e11a462261413eb85b4 [id=118053de49694cac881977b65f70b182]) successfully deleted
I20251211 08:57:30.665596 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 62) took 1183 ms. Trace:
I20251211 08:57:30.665906 14718 rpcz_store.cc:276] 1211 08:57:29.481890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:29.543482 (+ 61592us) service_pool.cc:225] Handling call
1211 08:57:29.608295 (+ 64813us) ts_tablet_manager.cc:1151] Deleting tablet 7d4f6727c4464cc9b7d68866c3969a0f
1211 08:57:29.608338 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:30.192590 (+584252us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:30.560088 (+367498us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:30.665517 (+105429us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":148202,"tablet-delete.queue_time_us":62915}
I20251211 08:57:30.667757 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 7d4f6727c4464cc9b7d68866c3969a0f (table default.Table_c76caa256ffd4898a64d730f6583004f [id=e6bc81a67ba949c69798c5049da505b5]) successfully deleted
I20251211 08:57:30.668476 14905 ts_tablet_manager.cc:1916] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:30.668855 14718 tablet_replica.cc:333] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:30.669502 14718 raft_consensus.cc:2243] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:30.669945 14718 raft_consensus.cc:2272] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:30.672037 14718 ts_tablet_manager.cc:1916] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:30.673427 14703 tablet_bootstrap.cc:492] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:30.673818 14703 ts_tablet_manager.cc:1403] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.160s user 0.015s sys 0.016s
I20251211 08:57:30.676045 14703 raft_consensus.cc:359] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.676816 14703 raft_consensus.cc:385] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.677127 14703 raft_consensus.cc:740] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.683096 14703 consensus_queue.cc:260] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.700896 14703 ts_tablet_manager.cc:1434] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20251211 08:57:30.690688 14714 tablet_bootstrap.cc:654] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.704636 14703 tablet_bootstrap.cc:492] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:30.715720 14904 raft_consensus.cc:2243] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:30.725243 14904 pending_rounds.cc:70] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
I20251211 08:57:30.725474 14904 pending_rounds.cc:77] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256372692385792 op_type: NO_OP noop_request { }
I20251211 08:57:30.725816 14904 raft_consensus.cc:2889] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:30.726117 14904 raft_consensus.cc:2272] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:30.729689 14703 tablet_bootstrap.cc:654] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.731509 14008 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet d2a389d6eb0f4576b5565eea70979229
I20251211 08:57:30.737857 14005 catalog_manager.cc:5654] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:30.739586 14005 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 426e26859b9c4966b8ec8ba3eab53a43 (table default.Table_a144e458c8554442a03170f402a13d01 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 1 got 0
W20251211 08:57:30.742305 14986 consensus_queue.cc:1175] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "31404d5f58e548519239569b150d975b" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20251211 08:57:30.744803 14922 raft_consensus.cc:493] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:30.745407 14722 tablet_bootstrap.cc:492] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:30.745213 14922 raft_consensus.cc:515] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.745754 14722 ts_tablet_manager.cc:1403] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.585s user 0.100s sys 0.200s
I20251211 08:57:30.746968 14922 leader_election.cc:290] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:30.747871 14722 raft_consensus.cc:359] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.748414 14722 raft_consensus.cc:385] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.748441 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559f9d428b5841708999725b5b859660" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:30.748658 14722 raft_consensus.cc:740] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.749133 14446 raft_consensus.cc:2393] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate c5e624bcfd154cd586ef1dedb9940063 in this term.
I20251211 08:57:30.749231 14722 consensus_queue.cc:260] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.751106 14722 ts_tablet_manager.cc:1434] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251211 08:57:30.751605 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559f9d428b5841708999725b5b859660" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:30.751900 14722 tablet_bootstrap.cc:492] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:30.753440 14230 leader_election.cc:304] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:30.757426 14904 ts_tablet_manager.cc:1916] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:30.757661 14922 raft_consensus.cc:3060] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:30.760300 14703 tablet_bootstrap.cc:492] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:30.760690 14703 ts_tablet_manager.cc:1403] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.004s
I20251211 08:57:30.762323 14722 tablet_bootstrap.cc:654] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.763276 14703 raft_consensus.cc:359] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.763868 14703 raft_consensus.cc:385] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.764104 14703 raft_consensus.cc:740] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.764760 14703 consensus_queue.cc:260] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.765800 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_0551d84bbd1b41399aab17ec5d23007c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:30.767338 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0551d84bbd1b41399aab17ec5d23007c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:30.771000 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_02e79cde30df4a2d8e81657161c2a56e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:30.772620 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02e79cde30df4a2d8e81657161c2a56e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:30.782582 14714 tablet_bootstrap.cc:492] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:30.783025 14714 ts_tablet_manager.cc:1403] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.124s user 0.007s sys 0.006s
I20251211 08:57:30.785648 14714 raft_consensus.cc:359] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.786221 14714 raft_consensus.cc:385] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.786489 14714 raft_consensus.cc:740] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.788545 14703 ts_tablet_manager.cc:1434] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
I20251211 08:57:30.788638 14722 tablet_bootstrap.cc:492] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:30.788976 14722 ts_tablet_manager.cc:1403] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.000s
I20251211 08:57:30.790701 14703 tablet_bootstrap.cc:492] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:30.791234 14722 raft_consensus.cc:359] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.791877 14722 raft_consensus.cc:385] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.792171 14722 raft_consensus.cc:740] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.796066 14703 tablet_bootstrap.cc:654] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.787063 14714 consensus_queue.cc:260] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.805613 14714 ts_tablet_manager.cc:1434] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20251211 08:57:30.806423 14714 tablet_bootstrap.cc:492] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:30.792783 14722 consensus_queue.cc:260] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.821899 14722 ts_tablet_manager.cc:1434] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20251211 08:57:30.822800 14722 tablet_bootstrap.cc:492] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:30.833184 14703 tablet_bootstrap.cc:492] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:30.833612 14703 ts_tablet_manager.cc:1403] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.000s
I20251211 08:57:30.836119 14703 raft_consensus.cc:359] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.836791 14703 raft_consensus.cc:385] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.837069 14703 raft_consensus.cc:740] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.837680 14703 consensus_queue.cc:260] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.838203 14714 tablet_bootstrap.cc:654] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.842943 14722 tablet_bootstrap.cc:654] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:30.846452 14703 ts_tablet_manager.cc:1434] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20251211 08:57:30.856226 14714 tablet_bootstrap.cc:492] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:30.856639 14714 ts_tablet_manager.cc:1403] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.000s
I20251211 08:57:30.858589 14714 raft_consensus.cc:359] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.859208 14714 raft_consensus.cc:385] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:30.859436 14714 raft_consensus.cc:740] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:30.859936 14714 consensus_queue.cc:260] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.864097 14922 raft_consensus.cc:2749] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:30.866631 14905 ts_tablet_manager.cc:1929] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:30.867043 14905 log.cc:1199] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/fab1ee5cf4eb4e11a72f5ad1de8c3856
I20251211 08:57:30.867599 14905 ts_tablet_manager.cc:1950] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:57:30.874287 14004 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2cd6e4352cfc41e497dc58d158e24ff2 has no consensus state
I20251211 08:57:30.875798 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 45f3395a34cf4fdd8e1898b08d123c8a
I20251211 08:57:30.876914 14013 catalog_manager.cc:6724] ReplaceTablet: tablet 45f3395a34cf4fdd8e1898b08d123c8a deleted and replaced by tablet afe339c4cc95490683618e18a3da252b
I20251211 08:57:30.877813 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet 45f3395a34cf4fdd8e1898b08d123c8a with delete_type TABLET_DATA_DELETED (replaced by tablet afe339c4cc95490683618e18a3da252b) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:30.880545 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 45f3395a34cf4fdd8e1898b08d123c8a with delete_type TABLET_DATA_DELETED (replaced by tablet afe339c4cc95490683618e18a3da252b) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:30.882035 14714 ts_tablet_manager.cc:1434] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20251211 08:57:30.885535 14714 tablet_bootstrap.cc:492] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:30.886011 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 45f3395a34cf4fdd8e1898b08d123c8a with delete_type TABLET_DATA_DELETED (replaced by tablet afe339c4cc95490683618e18a3da252b) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:30.897121 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7422c10fa9f240fe82576a2917407bef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:30.898713 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7422c10fa9f240fe82576a2917407bef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:30.900947 14714 tablet_bootstrap.cc:654] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:30.920068 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 166769us
I20251211 08:57:30.925593 14960 raft_consensus.cc:493] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:30.926010 14960 raft_consensus.cc:515] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.927841 14960 leader_election.cc:290] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:30.930092 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15f7bcd680064b9f8fe51aa05acce6fb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:30.932102 14542 leader_election.cc:343] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.932830 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15f7bcd680064b9f8fe51aa05acce6fb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:30.935539 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4e029d83ecc947918a83e94beb8871c5" } modify_external_catalogs: true
I20251211 08:57:30.936098 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4e029d83ecc947918a83e94beb8871c5" } modify_external_catalogs: true
W20251211 08:57:30.940001 14544 leader_election.cc:343] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.940342 14544 leader_election.cc:304] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:30.941208 14881 raft_consensus.cc:2749] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:30.957392 14005 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 31) took 807 ms (client timeout 1000 ms). Trace:
I20251211 08:57:30.960471 14718 ts_tablet_manager.cc:1929] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:30.960515 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 1a2dcb9ac62e4aac885d638f4e2b072e (table default.Table_4550b341c9cc4c82b83b4daf817232bd [id=1673af33c2f846c7bf092303eff6e8ad]). Expected version 2 got 1
I20251211 08:57:30.961853 14718 log.cc:1199] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/fab1ee5cf4eb4e11a72f5ad1de8c3856
I20251211 08:57:30.961925 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 1853070f81c34b52b8c74a7402e6dad3 (table default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4]). Expected version 1 got 0
I20251211 08:57:30.962931 14718 ts_tablet_manager.cc:1950] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:30.965404 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 38655ee4996e41cc8f87b5248264613d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:30.965988 14274 tablet_service.cc:1552] Processing DeleteTablet for tablet c5191443773b415198687c55c4517692 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:30.966902 14905 tablet_replica.cc:333] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:30.962751 14012 catalog_manager.cc:5654] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:30.967682 14905 raft_consensus.cc:2243] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:30.967713 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet a0f9c321e673433498a9059dd2e2efe6 (table default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5]). Expected version 1 got 0
I20251211 08:57:30.968168 14905 raft_consensus.cc:2272] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:30.968361 14012 catalog_manager.cc:5654] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:30.968941 14881 consensus_queue.cc:1048] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:30.970095 14905 ts_tablet_manager.cc:1916] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:30.971364 14904 ts_tablet_manager.cc:1929] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:30.971751 14904 log.cc:1199] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/fab1ee5cf4eb4e11a72f5ad1de8c3856
I20251211 08:57:30.972548 14904 ts_tablet_manager.cc:1950] T fab1ee5cf4eb4e11a72f5ad1de8c3856 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:30.973487 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 (table default.Table_1ab8b3b3713f428293902db0aa8cedb6 [id=67d1517b43294838bda420b792edb9c5]) successfully deleted
W20251211 08:57:30.957751 14005 rpcz_store.cc:269] 1211 08:57:30.149626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.149738 (+ 112us) service_pool.cc:225] Handling call
1211 08:57:30.403196 (+253458us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:30.957320 (+554124us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:30.741825 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:30.742175 (+ 350us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:30.742198 (+ 23us) write_op.cc:435] Acquired schema lock
1211 08:57:30.742217 (+ 19us) tablet.cc:660] Decoding operations
1211 08:57:30.742475 (+ 258us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:30.742525 (+ 50us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:30.742547 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:30.742679 (+ 132us) tablet.cc:699] Row locks acquired
1211 08:57:30.742692 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:30.742929 (+ 237us) write_op.cc:270] Start()
1211 08:57:30.743006 (+ 77us) write_op.cc:276] Timestamp: P: 1765443450743008 usec, L: 0
1211 08:57:30.743021 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:57:30.770919 (+ 27898us) log.cc:844] Serialized 530 byte log entry
1211 08:57:30.771498 (+ 579us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbef5a1 00007fc20bc81aa8 00007fc20c0f34d1 00007fc20c0f2ca8 00007fc20c0f7941 00007fc20c0f78f9 00007fc20c0f7889 00007fc20c0f7851 00007fc20c0f6a7d 00000000005aa2b4 00000000005aa0e9
1211 08:57:30.949555 (+178057us) op_driver.cc:464] REPLICATION: finished
1211 08:57:30.950397 (+ 842us) write_op.cc:301] APPLY: starting
1211 08:57:30.950451 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:30.950535 (+ 84us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:30.950550 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:30.950689 (+ 139us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:30.950841 (+ 152us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:30.950883 (+ 42us) write_op.cc:312] APPLY: finished
1211 08:57:30.951034 (+ 151us) log.cc:844] Serialized 23 byte log entry
1211 08:57:30.951171 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:30.951235 (+ 64us) write_op.cc:454] Released schema lock
1211 08:57:30.951262 (+ 27us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:30.951485 (+ 223us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:30.951507 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":534536448,"child_traces":[["op",{"apply.queue_time_us":693,"apply.run_cpu_time_us":1064,"apply.run_wall_time_us":1063,"num_ops":1,"prepare.queue_time_us":316,"prepare.run_cpu_time_us":2592,"prepare.run_wall_time_us":29797,"replication_time_us":206446,"spinlock_wait_cycles":62974848}]]}
I20251211 08:57:30.975883 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 (table default.Table_1ab8b3b3713f428293902db0aa8cedb6 [id=67d1517b43294838bda420b792edb9c5]) successfully deleted
W20251211 08:57:30.976639 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet e015d0fb9d6c4c80a510c05244b334e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e015d0fb9d6c4c80a510c05244b334e4 already in progress: creating tablet
I20251211 08:57:30.992637 14881 raft_consensus.cc:493] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:30.993099 14881 raft_consensus.cc:515] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:30.995983 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25805a95349d49f58280a30e6876929d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:30.997201 14542 leader_election.cc:343] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:30.996011 14881 leader_election.cc:290] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
W20251211 08:57:31.000213 14382 consensus_peers.cc:597] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b60d99c2b4f442da4a4d9c5190f5797. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:31.016824 14722 tablet_bootstrap.cc:492] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:31.017216 14722 ts_tablet_manager.cc:1403] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.195s user 0.011s sys 0.012s
I20251211 08:57:31.019240 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25805a95349d49f58280a30e6876929d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:31.021777 14544 leader_election.cc:343] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:31.022327 14544 leader_election.cc:304] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:31.023167 15012 raft_consensus.cc:2749] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:31.027604 15012 consensus_queue.cc:1048] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:31.031060 14714 tablet_bootstrap.cc:492] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:31.031584 14714 ts_tablet_manager.cc:1403] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.149s user 0.011s sys 0.000s
I20251211 08:57:31.034137 14714 raft_consensus.cc:359] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.034551 14997 raft_consensus.cc:493] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:31.034759 14714 raft_consensus.cc:385] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:31.035023 14714 raft_consensus.cc:740] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:31.035081 14997 raft_consensus.cc:515] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.035557 14714 consensus_queue.cc:260] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.037431 14714 ts_tablet_manager.cc:1434] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251211 08:57:31.038057 14997 leader_election.cc:290] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:31.038254 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b9b338a3f544ccb813247250b0f0eb7" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:31.019397 14722 raft_consensus.cc:359] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.040023 14722 raft_consensus.cc:385] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:31.040769 14722 raft_consensus.cc:740] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
W20251211 08:57:31.046365 14382 consensus_peers.cc:597] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:31.049443 14718 tablet_replica.cc:333] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:31.050361 14718 raft_consensus.cc:2243] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:31.051420 14718 raft_consensus.cc:2272] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:31.060710 14007 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 39796e6d0dc44b5489724d01e8270d05
W20251211 08:57:31.070065 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2cd6e4352cfc41e497dc58d158e24ff2 (table default.Table_9c85250afba741a79335663257915c82 [id=48f11a87431349f8a3caeb458f06869d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2cd6e4352cfc41e497dc58d158e24ff2 has no consensus state
I20251211 08:57:31.078301 14722 consensus_queue.cc:260] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.081252 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a144e458c8554442a03170f402a13d01" } new_table_name: "default.Table_4f6449beeb5d4b80ac26c12176d3d209" modify_external_catalogs: true
I20251211 08:57:31.086228 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_116c9891f83d47438f27b42312aa4cce" } new_table_name: "default.Table_f73c2258efaf4858800af6ac28f6d3a1" modify_external_catalogs: true
I20251211 08:57:31.093881 14718 ts_tablet_manager.cc:1916] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:31.094934 15011 tablet.cc:1722] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251211 08:57:31.040009 14231 leader_election.cc:343] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:31.106850 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 426e26859b9c4966b8ec8ba3eab53a43 consensus state has no leader
W20251211 08:57:31.117102 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 77156.8us
I20251211 08:57:31.121945 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_d12516554d1547419ff004d016cb9a3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:31.123453 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d12516554d1547419ff004d016cb9a3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:31.130190 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 (table default.Table_1ab8b3b3713f428293902db0aa8cedb6 [id=67d1517b43294838bda420b792edb9c5]) successfully deleted
I20251211 08:57:31.118642 15013 tablet.cc:1722] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251211 08:57:31.137956 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet e015d0fb9d6c4c80a510c05244b334e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e015d0fb9d6c4c80a510c05244b334e4 already in progress: creating tablet
I20251211 08:57:31.118949 14904 tablet_replica.cc:333] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:31.142359 14904 raft_consensus.cc:2243] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:31.158879 14722 ts_tablet_manager.cc:1434] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.141s user 0.008s sys 0.000s
I20251211 08:57:31.178197 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_9684313a6b2744babaa1226244fcbb94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:31.179800 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9684313a6b2744babaa1226244fcbb94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:31.182339 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b535219fa3ff4e8db580c8ab13bb1bc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:31.183724 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b535219fa3ff4e8db580c8ab13bb1bc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:31.185344 14009 master_service.cc:954] ReplaceTablet: received request to replace tablet 00f79094ed8b4782a3a1488e41794f9b from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:31.220654 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet a0f9c321e673433498a9059dd2e2efe6 (table default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5]). Expected version 1 got 0
I20251211 08:57:31.159888 14961 raft_consensus.cc:493] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:31.226509 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 9168007ca0584954a734e2c8f2e73f70 (DEFAULT_TABLE table=default.Table_b997f016aa90456c8c6b1a4dcf5f4c75 [id=dece67928a044390a750633247cfc2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.226958 14961 raft_consensus.cc:515] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.228773 14961 leader_election.cc:290] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:31.163779 14904 raft_consensus.cc:2272] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:31.230878 14904 ts_tablet_manager.cc:1916] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:31.214432 14274 tablet_service.cc:1505] Processing CreateTablet for tablet ccce5a446ac9460c9c9b7de72d875d95 (DEFAULT_TABLE table=default.Table_afa03065224d44a3a321c94b59f12587 [id=6d0af44f2d4542168eb2af3ead7fe1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.233062 14961 raft_consensus.cc:493] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:31.233522 14961 raft_consensus.cc:515] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.235173 14961 leader_election.cc:290] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:31.240329 14722 tablet_bootstrap.cc:492] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:31.242978 14275 tablet_service.cc:1505] Processing CreateTablet for tablet eb5a6e533e1f4da7b03a800d029f712d (DEFAULT_TABLE table=default.Table_8dd853c2cd9f4a18816d6724773e96cd [id=1e865cf903a347eaa103484c3a486d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.252823 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 4e410da32b3446edbf84b2eaa96424c3 (DEFAULT_TABLE table=default.Table_eb447b641c624aeba08c478e373a532d [id=38f5e9afc13a4797a7e8460f1f073280]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.266098 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_28a52518794d42d49dac3912b531c37c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:31.267818 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28a52518794d42d49dac3912b531c37c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:31.278285 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b535219fa3ff4e8db580c8ab13bb1bc4" } modify_external_catalogs: true
I20251211 08:57:31.283607 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b535219fa3ff4e8db580c8ab13bb1bc4" } modify_external_catalogs: true
I20251211 08:57:31.297049 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9168007ca0584954a734e2c8f2e73f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.301337 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5a6e533e1f4da7b03a800d029f712d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.304766 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccce5a446ac9460c9c9b7de72d875d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.306241 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e410da32b3446edbf84b2eaa96424c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.306910 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 00f79094ed8b4782a3a1488e41794f9b (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]). Expected version 2 got 0
I20251211 08:57:31.309984 14004 catalog_manager.cc:5654] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:31.091349 15010 tablet.cc:1722] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:31.038265 14714 tablet_bootstrap.cc:492] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:31.187758 14385 consensus_peers.cc:597] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b60d99c2b4f442da4a4d9c5190f5797. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:31.331069 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 143487us
I20251211 08:57:31.189863 14449 raft_consensus.cc:3060] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:31.333491 14961 raft_consensus.cc:493] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:31.333921 14961 raft_consensus.cc:515] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.335602 14425 tablet_service.cc:1505] Processing CreateTablet for tablet ccce5a446ac9460c9c9b7de72d875d95 (DEFAULT_TABLE table=default.Table_afa03065224d44a3a321c94b59f12587 [id=6d0af44f2d4542168eb2af3ead7fe1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.336614 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12379255548f49dfad17f3840a0c86bb" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:31.337365 14961 leader_election.cc:290] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:31.337255 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 9168007ca0584954a734e2c8f2e73f70 (DEFAULT_TABLE table=default.Table_b997f016aa90456c8c6b1a4dcf5f4c75 [id=dece67928a044390a750633247cfc2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.336627 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccce5a446ac9460c9c9b7de72d875d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.338070 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9168007ca0584954a734e2c8f2e73f70. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:31.338888 14231 leader_election.cc:343] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:31.341116 14422 tablet_service.cc:1505] Processing CreateTablet for tablet eb5a6e533e1f4da7b03a800d029f712d (DEFAULT_TABLE table=default.Table_8dd853c2cd9f4a18816d6724773e96cd [id=1e865cf903a347eaa103484c3a486d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.342140 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5a6e533e1f4da7b03a800d029f712d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.344425 14423 tablet_service.cc:1505] Processing CreateTablet for tablet 4e410da32b3446edbf84b2eaa96424c3 (DEFAULT_TABLE table=default.Table_eb447b641c624aeba08c478e373a532d [id=38f5e9afc13a4797a7e8460f1f073280]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.345398 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e410da32b3446edbf84b2eaa96424c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.347421 14429 tablet_service.cc:1505] Processing CreateTablet for tablet 6de43c843ff845a0bd4eb98b6054dd87 (DEFAULT_TABLE table=default.Table_04406fa17cb54541b721e2b11e4b831a [id=e5ffdeadfa904d85b68d0925da7e7a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.348374 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de43c843ff845a0bd4eb98b6054dd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.238874 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00f79094ed8b4782a3a1488e41794f9b" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:31.239653 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2c6080b3c1e4897bc1881e50f29f3a0" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:31.351838 14231 leader_election.cc:343] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:31.352444 14231 leader_election.cc:343] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:31.361912 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 5cf1210733754f43885a7c0a350aa206 (DEFAULT_TABLE table=default.Table_a939f5e4faf546148710852f96ded5d8 [id=e9973d6c9b1341d1b96f04367e44e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.362959 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 (DEFAULT_TABLE table=default.Table_9c85250afba741a79335663257915c82 [id=48f11a87431349f8a3caeb458f06869d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.364907 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 3303dc521d694be6b386a35da1172ede (DEFAULT_TABLE table=default.Table_4d13dd70ed244e54861bb2debdf39715 [id=ebfd6e4105a8410bb79a6db704ec0ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.377774 14714 tablet_bootstrap.cc:654] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:31.383620 14938 raft_consensus.cc:493] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:31.384721 14938 raft_consensus.cc:515] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.386754 14938 leader_election.cc:290] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:31.388978 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38655ee4996e41cc8f87b5248264613d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:31.389611 14296 raft_consensus.cc:2468] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:31.391273 14385 leader_election.cc:304] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:31.399107 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf1210733754f43885a7c0a350aa206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.399691 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd6e4352cfc41e497dc58d158e24ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.401696 14581 tablet_service.cc:1505] Processing CreateTablet for tablet 3303dc521d694be6b386a35da1172ede (DEFAULT_TABLE table=default.Table_4d13dd70ed244e54861bb2debdf39715 [id=ebfd6e4105a8410bb79a6db704ec0ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.403280 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3303dc521d694be6b386a35da1172ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.403786 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3303dc521d694be6b386a35da1172ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.411974 14582 tablet_service.cc:1505] Processing CreateTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 (DEFAULT_TABLE table=default.Table_9c85250afba741a79335663257915c82 [id=48f11a87431349f8a3caeb458f06869d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.413328 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd6e4352cfc41e497dc58d158e24ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.413275 14938 raft_consensus.cc:2804] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:31.413715 14938 raft_consensus.cc:493] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:31.414052 14938 raft_consensus.cc:3060] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:31.417697 14961 consensus_queue.cc:1048] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:31.421489 14583 tablet_service.cc:1505] Processing CreateTablet for tablet ccce5a446ac9460c9c9b7de72d875d95 (DEFAULT_TABLE table=default.Table_afa03065224d44a3a321c94b59f12587 [id=6d0af44f2d4542168eb2af3ead7fe1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.422920 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccce5a446ac9460c9c9b7de72d875d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.432014 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 4e410da32b3446edbf84b2eaa96424c3 (DEFAULT_TABLE table=default.Table_eb447b641c624aeba08c478e373a532d [id=38f5e9afc13a4797a7e8460f1f073280]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.433491 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e410da32b3446edbf84b2eaa96424c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.440644 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 (DEFAULT_TABLE table=default.Table_9c85250afba741a79335663257915c82 [id=48f11a87431349f8a3caeb458f06869d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.443360 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 5cf1210733754f43885a7c0a350aa206 (DEFAULT_TABLE table=default.Table_a939f5e4faf546148710852f96ded5d8 [id=e9973d6c9b1341d1b96f04367e44e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.444905 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 6de43c843ff845a0bd4eb98b6054dd87 (DEFAULT_TABLE table=default.Table_04406fa17cb54541b721e2b11e4b831a [id=e5ffdeadfa904d85b68d0925da7e7a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.446480 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 3303dc521d694be6b386a35da1172ede (DEFAULT_TABLE table=default.Table_4d13dd70ed244e54861bb2debdf39715 [id=ebfd6e4105a8410bb79a6db704ec0ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.447933 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 5cf1210733754f43885a7c0a350aa206 (DEFAULT_TABLE table=default.Table_a939f5e4faf546148710852f96ded5d8 [id=e9973d6c9b1341d1b96f04367e44e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.449003 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf1210733754f43885a7c0a350aa206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.451601 14586 tablet_service.cc:1505] Processing CreateTablet for tablet eb5a6e533e1f4da7b03a800d029f712d (DEFAULT_TABLE table=default.Table_8dd853c2cd9f4a18816d6724773e96cd [id=1e865cf903a347eaa103484c3a486d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.452679 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5a6e533e1f4da7b03a800d029f712d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.454435 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 9168007ca0584954a734e2c8f2e73f70 (DEFAULT_TABLE table=default.Table_b997f016aa90456c8c6b1a4dcf5f4c75 [id=dece67928a044390a750633247cfc2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.455293 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9168007ca0584954a734e2c8f2e73f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.457012 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 6de43c843ff845a0bd4eb98b6054dd87 (DEFAULT_TABLE table=default.Table_04406fa17cb54541b721e2b11e4b831a [id=e5ffdeadfa904d85b68d0925da7e7a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:31.457846 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de43c843ff845a0bd4eb98b6054dd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.479293 14005 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet cda5088295e048e9be8e72cffaaa3e30
I20251211 08:57:31.480082 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4d13dd70ed244e54861bb2debdf39715" } new_table_name: "default.Table_a6e8f55c1ec74e60b16628cac13435f5" modify_external_catalogs: true
I20251211 08:57:31.488165 14862 raft_consensus.cc:493] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:31.488479 14961 consensus_queue.cc:1048] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:31.488660 14862 raft_consensus.cc:515] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.490494 14862 leader_election.cc:290] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:31.493263 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5191443773b415198687c55c4517692" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:31.494081 14296 raft_consensus.cc:2393] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate c5e624bcfd154cd586ef1dedb9940063 in this term.
I20251211 08:57:31.498176 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3303dc521d694be6b386a35da1172ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.498737 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd6e4352cfc41e497dc58d158e24ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.503180 14904 ts_tablet_manager.cc:1929] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:31.314606 14722 tablet_bootstrap.cc:654] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:31.503959 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf1210733754f43885a7c0a350aa206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.506160 14904 log.cc:1199] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/53fc34c41afe44678b8d6dcf79a31f8b
I20251211 08:57:31.509398 14904 ts_tablet_manager.cc:1950] T 53fc34c41afe44678b8d6dcf79a31f8b P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:31.511579 14881 raft_consensus.cc:493] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:31.512373 14881 raft_consensus.cc:515] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:31.513193 14608 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.002s sys 0.010s
W20251211 08:57:31.513643 14608 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.525s user 0.002s sys 0.010s
I20251211 08:57:31.515130 14881 leader_election.cc:290] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:31.516134 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ccc5d49e4a09446bb4897be97093af29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:31.517740 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccc5d49e4a09446bb4897be97093af29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:31.519217 14881 raft_consensus.cc:493] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:31.519644 14881 raft_consensus.cc:515] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.521159 14881 leader_election.cc:290] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:31.541432 14714 tablet_bootstrap.cc:492] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:31.541945 14714 ts_tablet_manager.cc:1403] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.504s user 0.009s sys 0.003s
I20251211 08:57:31.566242 14714 raft_consensus.cc:359] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.566918 14714 raft_consensus.cc:385] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:31.568836 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet a0f9c321e673433498a9059dd2e2efe6 (table default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5]). Expected version 1 got 0
I20251211 08:57:31.584800 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 66) took 1465 ms. Trace:
I20251211 08:57:31.508004 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de43c843ff845a0bd4eb98b6054dd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:31.538393 14905 ts_tablet_manager.cc:1929] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20251211 08:57:31.574148 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3303dc521d694be6b386a35da1172ede consensus state has no leader
I20251211 08:57:31.597303 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_fcf104ee591d470d806156189c2cac52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:31.598724 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcf104ee591d470d806156189c2cac52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:31.603796 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 479d7fe06a284234aef420e3842f0eab (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]). Expected version 1 got 0
I20251211 08:57:31.574101 14938 raft_consensus.cc:515] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.623993 14938 leader_election.cc:290] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:31.626197 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38655ee4996e41cc8f87b5248264613d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:31.626669 14296 raft_consensus.cc:3060] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:31.627149 14005 master_service.cc:954] ReplaceTablet: received request to replace tablet 60b23163906447c3877db631968f5623 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:31.630681 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 426e26859b9c4966b8ec8ba3eab53a43 (table default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 2 got 0
I20251211 08:57:31.596314 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 00f79094ed8b4782a3a1488e41794f9b
I20251211 08:57:31.633122 14279 tablet_service.cc:1552] Processing DeleteTablet for tablet 00f79094ed8b4782a3a1488e41794f9b with delete_type TABLET_DATA_DELETED (replaced by tablet 78c6a703e1e44f41931fe465def01bf6) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:31.633875 14009 catalog_manager.cc:6724] ReplaceTablet: tablet 00f79094ed8b4782a3a1488e41794f9b deleted and replaced by tablet 78c6a703e1e44f41931fe465def01bf6
I20251211 08:57:31.595736 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_e3e740f7ffee4b1683b5888496b1a384"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:31.636377 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3e740f7ffee4b1683b5888496b1a384 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:31.642700 14904 rpcz_store.cc:276] 1211 08:57:30.119417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.121279 (+ 1862us) service_pool.cc:225] Handling call
1211 08:57:31.118804 (+997525us) ts_tablet_manager.cc:1151] Deleting tablet 53fc34c41afe44678b8d6dcf79a31f8b
1211 08:57:31.118838 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:31.373081 (+254243us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.502876 (+129795us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.584738 (+ 81862us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20768,"tablet-delete.queue_time_us":997021}
I20251211 08:57:31.648794 14422 tablet_service.cc:1552] Processing DeleteTablet for tablet 00f79094ed8b4782a3a1488e41794f9b with delete_type TABLET_DATA_DELETED (replaced by tablet 78c6a703e1e44f41931fe465def01bf6) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:31.567198 14714 raft_consensus.cc:740] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
W20251211 08:57:31.315837 15011 tablet_replica.cc:1410] Time spent flushing the Log queue took a long time: real 0.216s user 0.001s sys 0.000s
I20251211 08:57:31.659014 14714 consensus_queue.cc:260] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.661526 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 60b23163906447c3877db631968f5623
I20251211 08:57:31.658862 14905 log.cc:1199] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/53fc34c41afe44678b8d6dcf79a31f8b
I20251211 08:57:31.670477 14714 ts_tablet_manager.cc:1434] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.128s user 0.007s sys 0.000s
I20251211 08:57:31.678668 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 00f79094ed8b4782a3a1488e41794f9b with delete_type TABLET_DATA_DELETED (replaced by tablet 78c6a703e1e44f41931fe465def01bf6) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:31.680130 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 60b23163906447c3877db631968f5623 with delete_type TABLET_DATA_DELETED (replaced by tablet e961c5a3b5f640cbaea78b6185048d04) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:31.680457 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet a0f9c321e673433498a9059dd2e2efe6 (table default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5]). Expected version 1 got 0
I20251211 08:57:31.683204 14714 tablet_bootstrap.cc:492] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:31.685513 14009 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9c85250afba741a79335663257915c82" } modify_external_catalogs: true
I20251211 08:57:31.686115 14009 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9c85250afba741a79335663257915c82" } modify_external_catalogs: true
W20251211 08:57:31.660671 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 232604us
I20251211 08:57:31.691426 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 53fc34c41afe44678b8d6dcf79a31f8b (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]) successfully deleted
I20251211 08:57:31.692162 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00f79094ed8b4782a3a1488e41794f9b" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:31.692922 14603 raft_consensus.cc:2468] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:31.648241 14722 tablet_bootstrap.cc:492] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:31.672784 14296 raft_consensus.cc:2468] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:31.677830 14279 tablet_service.cc:1552] Processing DeleteTablet for tablet 60b23163906447c3877db631968f5623 with delete_type TABLET_DATA_DELETED (replaced by tablet e961c5a3b5f640cbaea78b6185048d04) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:31.695772 14385 leader_election.cc:304] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:31.696070 14722 ts_tablet_manager.cc:1403] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.459s user 0.012s sys 0.000s
I20251211 08:57:31.696581 14938 raft_consensus.cc:2804] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:31.696938 14938 raft_consensus.cc:697] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:31.697597 14938 consensus_queue.cc:237] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.698156 14005 catalog_manager.cc:6724] ReplaceTablet: tablet 60b23163906447c3877db631968f5623 deleted and replaced by tablet e961c5a3b5f640cbaea78b6185048d04
I20251211 08:57:31.698637 14722 raft_consensus.cc:359] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.699250 14722 raft_consensus.cc:385] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:31.699476 14722 raft_consensus.cc:740] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:31.705076 14905 ts_tablet_manager.cc:1950] T 53fc34c41afe44678b8d6dcf79a31f8b P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:31.681813 14008 catalog_manager.cc:5654] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:31.705789 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 426e26859b9c4966b8ec8ba3eab53a43 (table default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 2 got 0
I20251211 08:57:31.704201 14904 tablet_replica.cc:333] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:31.707661 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 1853070f81c34b52b8c74a7402e6dad3
I20251211 08:57:31.708946 14006 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55614 (request call id 120) took 1436 ms. Trace:
I20251211 08:57:31.709761 14904 raft_consensus.cc:2243] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:31.710253 14904 raft_consensus.cc:2272] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:31.710209 14602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12379255548f49dfad17f3840a0c86bb" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:31.711534 14602 raft_consensus.cc:2468] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:31.712103 14904 ts_tablet_manager.cc:1916] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:31.709226 14006 rpcz_store.cc:276] 1211 08:57:30.272951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.377638 (+104687us) service_pool.cc:225] Handling call
1211 08:57:30.378896 (+ 1258us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:30.406547 (+ 27651us) catalog_manager.cc:2815] Modifying in-memory table state
1211 08:57:30.406722 (+ 175us) catalog_manager.cc:2824] Locking tablets
1211 08:57:31.067939 (+661217us) catalog_manager.cc:2838] Removing table and tablets from system table
1211 08:57:31.088353 (+ 20414us) catalog_manager.cc:2856] Removing table from by-name map
1211 08:57:31.088553 (+ 200us) catalog_manager.cc:2872] Committing in-memory state
1211 08:57:31.707574 (+619021us) catalog_manager.cc:2876] Aborting table tasks
1211 08:57:31.708895 (+ 1321us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:31.069871 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:31.070272 (+ 401us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:31.070295 (+ 23us) write_op.cc:435] Acquired schema lock
1211 08:57:31.070307 (+ 12us) tablet.cc:660] Decoding operations
1211 08:57:31.070589 (+ 282us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:31.070621 (+ 32us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:31.070641 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:31.070908 (+ 267us) tablet.cc:699] Row locks acquired
1211 08:57:31.070924 (+ 16us) write_op.cc:260] PREPARE: finished
1211 08:57:31.071014 (+ 90us) write_op.cc:270] Start()
1211 08:57:31.071081 (+ 67us) write_op.cc:276] Timestamp: P: 1765443451071016 usec, L: 0
1211 08:57:31.071095 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:31.071510 (+ 415us) log.cc:844] Serialized 793 byte log entry
1211 08:57:31.085814 (+ 14304us) op_driver.cc:464] REPLICATION: finished
1211 08:57:31.086843 (+ 1029us) write_op.cc:301] APPLY: starting
1211 08:57:31.086893 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:31.086985 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:31.087068 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:31.087299 (+ 231us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:31.087428 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:31.087466 (+ 38us) write_op.cc:312] APPLY: finished
1211 08:57:31.087618 (+ 152us) log.cc:844] Serialized 29 byte log entry
1211 08:57:31.087781 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:31.087871 (+ 90us) write_op.cc:454] Released schema lock
1211 08:57:31.087913 (+ 42us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:31.088208 (+ 295us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:31.088242 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":862,"apply.run_cpu_time_us":1291,"apply.run_wall_time_us":1291,"mutex_wait_us":112,"num_ops":2,"prepare.queue_time_us":93,"prepare.run_cpu_time_us":1887,"prepare.run_wall_time_us":1985,"raft.queue_time_us":1021,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":13,"replication_time_us":14641}]]}
I20251211 08:57:31.715014 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 1853070f81c34b52b8c74a7402e6dad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:30 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:31.715199 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 60b23163906447c3877db631968f5623 with delete_type TABLET_DATA_DELETED (replaced by tablet e961c5a3b5f640cbaea78b6185048d04) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:31.719111 14012 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_116c9891f83d47438f27b42312aa4cce [id=ab93470ba8f54070be718c240d49aa13] alter complete (version 2)
I20251211 08:57:31.728098 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 1853070f81c34b52b8c74a7402e6dad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:30 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:31.713290 14279 tablet_service.cc:1552] Processing DeleteTablet for tablet 1853070f81c34b52b8c74a7402e6dad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:30 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:31.720451 14722 consensus_queue.cc:260] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.730307 14722 ts_tablet_manager.cc:1434] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.034s user 0.004s sys 0.000s
I20251211 08:57:31.738191 15010 tablet.cc:1722] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:31.717294 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2c6080b3c1e4897bc1881e50f29f3a0" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:31.739133 14604 raft_consensus.cc:2468] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:31.738474 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b60d99c2b4f442da4a4d9c5190f5797 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:31.739653 15013 tablet.cc:1722] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:31.742363 14714 tablet_bootstrap.cc:654] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:31.745100 14012 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 20) took 1755 ms (client timeout 1000 ms). Trace:
I20251211 08:57:31.713667 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b9b338a3f544ccb813247250b0f0eb7" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:31.751251 14608 raft_consensus.cc:2468] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:31.751242 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7b8ca0c73d454d97a2aee4c1d08adece"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:31.752823 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b8ca0c73d454d97a2aee4c1d08adece in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:31.754591 14722 tablet_bootstrap.cc:492] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:31.758265 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 70) took 1727 ms. Trace:
I20251211 08:57:31.758574 14905 rpcz_store.cc:276] 1211 08:57:30.031153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.040784 (+ 9631us) service_pool.cc:225] Handling call
1211 08:57:30.966717 (+925933us) ts_tablet_manager.cc:1151] Deleting tablet 53fc34c41afe44678b8d6dcf79a31f8b
1211 08:57:30.966753 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:31.069620 (+102867us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.537911 (+468291us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.758202 (+220291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":925404}
I20251211 08:57:31.759747 14905 tablet_replica.cc:333] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:31.760572 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 53fc34c41afe44678b8d6dcf79a31f8b (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]) successfully deleted
I20251211 08:57:31.761368 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e354416047f74d02a9455147ca8d6bf3" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:31.772637 14722 tablet_bootstrap.cc:654] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:31.776476 14718 ts_tablet_manager.cc:1929] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:31.776791 14718 log.cc:1199] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/53fc34c41afe44678b8d6dcf79a31f8b
I20251211 08:57:31.777398 14718 ts_tablet_manager.cc:1950] T 53fc34c41afe44678b8d6dcf79a31f8b P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:31.778883 14905 raft_consensus.cc:2243] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:31.779361 14905 raft_consensus.cc:2272] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:31.781064 14905 ts_tablet_manager.cc:1916] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:31.781268 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 70) took 1639 ms. Trace:
I20251211 08:57:31.781533 14718 rpcz_store.cc:276] 1211 08:57:30.142155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.142485 (+ 330us) service_pool.cc:225] Handling call
1211 08:57:31.048545 (+906060us) ts_tablet_manager.cc:1151] Deleting tablet 53fc34c41afe44678b8d6dcf79a31f8b
1211 08:57:31.048588 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:31.570386 (+521798us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.776256 (+205870us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.781211 (+ 4955us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":904245}
I20251211 08:57:31.788082 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 53fc34c41afe44678b8d6dcf79a31f8b (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]) successfully deleted
I20251211 08:57:31.789808 14718 tablet_replica.cc:333] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:31.790647 14718 raft_consensus.cc:2243] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:31.791079 14718 raft_consensus.cc:2272] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:31.792604 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60b23163906447c3877db631968f5623" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:31.794247 14718 ts_tablet_manager.cc:1916] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:31.794355 14230 leader_election.cc:304] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:31.795025 14961 raft_consensus.cc:2804] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:31.795387 14961 raft_consensus.cc:493] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:31.795696 14961 raft_consensus.cc:3060] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:31.800426 14230 leader_election.cc:304] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:31.801155 14922 raft_consensus.cc:2804] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:31.801493 14922 raft_consensus.cc:493] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:31.801857 14922 raft_consensus.cc:3060] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:31.804620 14997 consensus_queue.cc:1048] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:31.805874 14997 consensus_queue.cc:1048] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:31.806547 14382 leader_election.cc:343] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 38655ee4996e41cc8f87b5248264613d
W20251211 08:57:31.808496 14382 leader_election.cc:343] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: c5191443773b415198687c55c4517692
I20251211 08:57:31.808828 14382 leader_election.cc:304] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:31.809720 14938 raft_consensus.cc:3060] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:31.816445 14382 leader_election.cc:343] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 38655ee4996e41cc8f87b5248264613d
I20251211 08:57:31.819936 14722 tablet_bootstrap.cc:492] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:31.820503 14722 ts_tablet_manager.cc:1403] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.000s
I20251211 08:57:31.820957 14714 tablet_bootstrap.cc:492] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:31.821352 14714 ts_tablet_manager.cc:1403] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.139s user 0.017s sys 0.013s
I20251211 08:57:31.822603 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e354416047f74d02a9455147ca8d6bf3" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:31.823549 14714 raft_consensus.cc:359] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.824088 14714 raft_consensus.cc:385] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:31.824295 14714 raft_consensus.cc:740] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
W20251211 08:57:31.824227 14542 leader_election.cc:343] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:31.824677 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60b23163906447c3877db631968f5623" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:31.828011 14542 leader_election.cc:343] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:31.829268 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4d0f3ca2b6c74400bc10c59a0f8f55f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:31.830983 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d0f3ca2b6c74400bc10c59a0f8f55f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:31.824792 14714 consensus_queue.cc:260] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.834779 14722 raft_consensus.cc:359] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.837643 14722 raft_consensus.cc:385] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:31.838580 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9684313a6b2744babaa1226244fcbb94" } new_table_name: "default.Table_0ba5761cf15c49bd81994a532279c459" modify_external_catalogs: true
I20251211 08:57:31.842025 14714 ts_tablet_manager.cc:1434] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20251211 08:57:31.842098 14722 raft_consensus.cc:740] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:31.844533 14722 consensus_queue.cc:260] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.845042 14714 tablet_bootstrap.cc:492] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:31.845290 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_5a729546e47b4fc3b53ef93b18b3e438"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:31.846799 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a729546e47b4fc3b53ef93b18b3e438 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:31.847128 14544 leader_election.cc:343] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:31.852134 14010 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_4550b341c9cc4c82b83b4daf817232bd [id=1673af33c2f846c7bf092303eff6e8ad] alter complete (version 2)
I20251211 08:57:31.853892 14714 tablet_bootstrap.cc:654] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:31.855161 14544 leader_election.cc:304] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:31.857208 14722 ts_tablet_manager.cc:1434] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.036s user 0.009s sys 0.000s
I20251211 08:57:31.857542 14960 raft_consensus.cc:2749] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:31.859827 14010 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 23) took 2042 ms (client timeout 1000 ms). Trace:
I20251211 08:57:31.861651 14581 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d4f6727c4464cc9b7d68866c3969a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:31.870869 14722 tablet_bootstrap.cc:492] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:31.873051 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 68378.1us
I20251211 08:57:31.876468 14722 tablet_bootstrap.cc:654] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:31.879608 14922 raft_consensus.cc:515] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.881392 14922 leader_election.cc:290] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:31.885262 14004 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 24) took 1053 ms (client timeout 1000 ms). Trace:
I20251211 08:57:31.887188 14961 raft_consensus.cc:515] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:31.885552 14004 rpcz_store.cc:269] 1211 08:57:30.831625 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.875091 (+ 43466us) service_pool.cc:225] Handling call
1211 08:57:30.902602 (+ 27511us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:31.554372 (+651770us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:31.554395 (+ 23us) write_op.cc:454] Released schema lock
1211 08:57:31.885194 (+330799us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:31.543205 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:31.543707 (+ 502us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:31.543740 (+ 33us) write_op.cc:435] Acquired schema lock
1211 08:57:31.543757 (+ 17us) tablet.cc:660] Decoding operations
1211 08:57:31.544042 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:31.544093 (+ 51us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:31.544133 (+ 40us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:31.544322 (+ 189us) tablet.cc:699] Row locks acquired
1211 08:57:31.544341 (+ 19us) write_op.cc:260] PREPARE: finished
1211 08:57:31.544473 (+ 132us) write_op.cc:270] Start()
1211 08:57:31.544575 (+ 102us) write_op.cc:276] Timestamp: P: 1765443451544312 usec, L: 0
1211 08:57:31.544596 (+ 21us) op_driver.cc:348] REPLICATION: starting
1211 08:57:31.545231 (+ 635us) log.cc:844] Serialized 530 byte log entry
1211 08:57:31.551560 (+ 6329us) op_driver.cc:464] REPLICATION: finished
1211 08:57:31.552012 (+ 452us) write_op.cc:301] APPLY: starting
1211 08:57:31.552065 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:31.552147 (+ 82us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:31.552235 (+ 88us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:31.552377 (+ 142us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:31.552497 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:31.552546 (+ 49us) write_op.cc:312] APPLY: finished
1211 08:57:31.552699 (+ 153us) log.cc:844] Serialized 23 byte log entry
1211 08:57:31.552822 (+ 123us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:31.552875 (+ 53us) write_op.cc:454] Released schema lock
1211 08:57:31.552903 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":18,"spinlock_wait_cycles":56097664,"child_traces":[["op",{"apply.queue_time_us":298,"apply.run_cpu_time_us":1045,"apply.run_wall_time_us":1044,"num_ops":1,"prepare.queue_time_us":31704,"prepare.run_cpu_time_us":2364,"prepare.run_wall_time_us":2376,"raft.queue_time_us":86,"raft.run_cpu_time_us":579,"raft.run_wall_time_us":578,"replication_time_us":6884,"spinlock_wait_cycles":36096}]]}
I20251211 08:57:31.888907 14961 leader_election.cc:290] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:31.890960 15013 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:31.895107 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12379255548f49dfad17f3840a0c86bb" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:31.896912 14922 raft_consensus.cc:493] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:31.897315 14922 raft_consensus.cc:515] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.900373 14922 leader_election.cc:290] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:31.901393 14230 leader_election.cc:304] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:31.903909 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00f79094ed8b4782a3a1488e41794f9b" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:31.904143 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e015d0fb9d6c4c80a510c05244b334e4" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:31.904552 14446 raft_consensus.cc:3060] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:31.905050 15010 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:31.907225 14718 ts_tablet_manager.cc:1929] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251211 08:57:31.907217 14231 leader_election.cc:343] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:31.916622 14231 leader_election.cc:343] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:31.917584 14230 raft_consensus.cc:2686] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251211 08:57:31.918325 14904 ts_tablet_manager.cc:1929] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:31.911744 14938 raft_consensus.cc:2749] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:31.920440 14718 log.cc:1199] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/d2c6080b3c1e4897bc1881e50f29f3a0
I20251211 08:57:31.876247 15030 tablet_bootstrap.cc:492] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:31.918392 14230 leader_election.cc:304] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
W20251211 08:57:31.862501 14544 leader_election.cc:343] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:31.926658 14904 log.cc:1199] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/d2c6080b3c1e4897bc1881e50f29f3a0
I20251211 08:57:31.927080 14544 leader_election.cc:304] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:31.927386 14904 ts_tablet_manager.cc:1950] T d2c6080b3c1e4897bc1881e50f29f3a0 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:31.927872 14960 raft_consensus.cc:2749] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:31.928643 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 66198.4us
I20251211 08:57:31.928256 15035 raft_consensus.cc:2804] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:31.929612 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 69) took 1808 ms. Trace:
I20251211 08:57:31.931407 14718 ts_tablet_manager.cc:1950] T d2c6080b3c1e4897bc1881e50f29f3a0 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:31.934278 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 73) took 1791 ms. Trace:
I20251211 08:57:31.929641 15035 raft_consensus.cc:493] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:31.935503 15035 raft_consensus.cc:3060] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:31.939857 14722 tablet_bootstrap.cc:492] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:31.934930 14904 rpcz_store.cc:276] 1211 08:57:30.121157 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.121353 (+ 196us) service_pool.cc:225] Handling call
1211 08:57:31.704042 (+1582689us) ts_tablet_manager.cc:1151] Deleting tablet d2c6080b3c1e4897bc1881e50f29f3a0
1211 08:57:31.704085 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:31.881933 (+177848us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.918107 (+ 36174us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.929553 (+ 11446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1582167}
I20251211 08:57:31.940205 14722 ts_tablet_manager.cc:1403] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.000s
I20251211 08:57:31.940696 14904 tablet_replica.cc:333] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:31.940761 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet d2c6080b3c1e4897bc1881e50f29f3a0 (table default.Table_86772d5552714163a6a0ffc236521623 [id=a5f4dfd6520045919362ac16b16d4acb]) successfully deleted
I20251211 08:57:31.941524 14904 raft_consensus.cc:2243] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:31.942051 14722 raft_consensus.cc:359] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.942378 14904 raft_consensus.cc:2272] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:31.942651 14722 raft_consensus.cc:385] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:31.942986 14722 raft_consensus.cc:740] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:31.946993 14722 consensus_queue.cc:260] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.944715 14714 tablet_bootstrap.cc:492] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:31.947047 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 53fc34c41afe44678b8d6dcf79a31f8b with delete_type TABLET_DATA_DELETED (replaced by tablet 383716a1f1f847ccbea542eca04790c9) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:31.934543 14446 raft_consensus.cc:2468] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:31.948465 14714 ts_tablet_manager.cc:1403] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.104s user 0.008s sys 0.004s
I20251211 08:57:31.949273 14231 leader_election.cc:304] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:31.950062 15038 raft_consensus.cc:2804] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:31.950481 15038 raft_consensus.cc:697] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:31.951256 15038 consensus_queue.cc:237] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.952342 14714 raft_consensus.cc:359] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.952919 14714 raft_consensus.cc:385] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:31.953166 14714 raft_consensus.cc:740] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:31.953651 14718 rpcz_store.cc:276] 1211 08:57:30.143078 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.189400 (+ 46322us) service_pool.cc:225] Handling call
1211 08:57:31.789610 (+1600210us) ts_tablet_manager.cc:1151] Deleting tablet d2c6080b3c1e4897bc1881e50f29f3a0
1211 08:57:31.789662 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:31.891139 (+101477us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.907012 (+ 15873us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.934215 (+ 27203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1584049}
I20251211 08:57:31.954664 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet d2c6080b3c1e4897bc1881e50f29f3a0 (table default.Table_86772d5552714163a6a0ffc236521623 [id=a5f4dfd6520045919362ac16b16d4acb]) successfully deleted
I20251211 08:57:31.955447 14904 ts_tablet_manager.cc:1916] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:31.957526 14722 ts_tablet_manager.cc:1434] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20251211 08:57:31.958400 14722 tablet_bootstrap.cc:492] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:31.960394 14718 tablet_replica.cc:333] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:31.961931 14718 raft_consensus.cc:2243] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:31.961897 15011 tablet.cc:1722] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:31.965561 15035 raft_consensus.cc:515] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.960836 14714 consensus_queue.cc:260] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:31.969065 14714 ts_tablet_manager.cc:1434] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20251211 08:57:31.966483 14718 raft_consensus.cc:2272] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:31.972301 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b9b338a3f544ccb813247250b0f0eb7" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:31.972574 14718 ts_tablet_manager.cc:1916] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:31.973465 14231 leader_election.cc:343] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:31.977355 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12379255548f49dfad17f3840a0c86bb" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:31.980443 15035 leader_election.cc:290] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:31.980561 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00f79094ed8b4782a3a1488e41794f9b" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:31.981053 14604 raft_consensus.cc:3060] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:31.985297 14722 tablet_bootstrap.cc:654] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:31.975771 14714 tablet_bootstrap.cc:492] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:31.990180 14604 raft_consensus.cc:2468] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:31.991935 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e015d0fb9d6c4c80a510c05244b334e4" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:31.993640 14714 tablet_bootstrap.cc:654] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:31.994673 14904 ts_tablet_manager.cc:1929] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:31.995113 14904 log.cc:1199] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/45f3395a34cf4fdd8e1898b08d123c8a
I20251211 08:57:31.995829 14904 ts_tablet_manager.cc:1950] T 45f3395a34cf4fdd8e1898b08d123c8a P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:31.996611 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 70) took 1116 ms. Trace:
I20251211 08:57:31.997110 14904 rpcz_store.cc:276] 1211 08:57:30.880256 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.880395 (+ 139us) service_pool.cc:225] Handling call
1211 08:57:31.940555 (+1060160us) ts_tablet_manager.cc:1151] Deleting tablet 45f3395a34cf4fdd8e1898b08d123c8a
1211 08:57:31.940593 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:31.985902 (+ 45309us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.994473 (+ 8571us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:31.996566 (+ 2093us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1059638}
I20251211 08:57:31.998724 14904 tablet_replica.cc:333] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:31.999481 14904 raft_consensus.cc:2243] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:31.999953 14904 raft_consensus.cc:2272] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:32.001775 14904 ts_tablet_manager.cc:1916] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:32.005713 15030 tablet_bootstrap.cc:654] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:31.962385 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ccc5d49e4a09446bb4897be97093af29" } modify_external_catalogs: true
I20251211 08:57:32.014678 14904 ts_tablet_manager.cc:1929] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:31.998382 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 45f3395a34cf4fdd8e1898b08d123c8a (table default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]) successfully deleted
I20251211 08:57:32.015141 14904 log.cc:1199] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/38655ee4996e41cc8f87b5248264613d
I20251211 08:57:32.015699 14904 ts_tablet_manager.cc:1950] T 38655ee4996e41cc8f87b5248264613d P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:31.983753 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet 15f7bcd680064b9f8fe51aa05acce6fb from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:32.016400 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 71) took 1052 ms. Trace:
I20251211 08:57:32.016806 14904 rpcz_store.cc:276] 1211 08:57:30.964247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.965233 (+ 986us) service_pool.cc:225] Handling call
1211 08:57:31.998589 (+1033356us) ts_tablet_manager.cc:1151] Deleting tablet 38655ee4996e41cc8f87b5248264613d
1211 08:57:31.998627 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:32.008507 (+ 9880us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.014490 (+ 5983us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.016355 (+ 1865us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1032822}
I20251211 08:57:32.018006 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 38655ee4996e41cc8f87b5248264613d (table default.Table_a7ade9d81bfc4498a43e0b5ad6941ac2 [id=c533026c25a54aa885eb483d4120f720]) successfully deleted
I20251211 08:57:32.000697 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 2cd6e4352cfc41e497dc58d158e24ff2
I20251211 08:57:32.018821 14904 tablet_replica.cc:333] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:32.020219 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:31 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:32.021669 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f988855c0a7d4baf90dab6f27a1ed140"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:32.021744 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:31 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:32.023041 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:31 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:32.019495 14904 raft_consensus.cc:2243] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:32.023788 14904 raft_consensus.cc:2272] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:32.025465 14904 ts_tablet_manager.cc:1916] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:32.027359 14600 raft_consensus.cc:3060] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:32.031587 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 40a1f12553934446bfe44f4ed41abf9a has no consensus state
I20251211 08:57:32.033474 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b9b338a3f544ccb813247250b0f0eb7" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:32.033944 14604 raft_consensus.cc:3060] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:32.033704 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_04406fa17cb54541b721e2b11e4b831a" } new_table_name: "default.Table_3c7e457251e74eb7b8415bbcb6b6d7db" modify_external_catalogs: true
W20251211 08:57:31.745967 14012 rpcz_store.cc:269] 1211 08:57:29.990182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.027054 (+ 36872us) service_pool.cc:225] Handling call
1211 08:57:30.095443 (+ 68389us) spinlock_profiling.cc:243] Waited 66 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:31.745023 (+1649580us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:30.980192 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:30.980607 (+ 415us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:30.980632 (+ 25us) write_op.cc:435] Acquired schema lock
1211 08:57:30.980646 (+ 14us) tablet.cc:660] Decoding operations
1211 08:57:30.980953 (+ 307us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:30.980990 (+ 37us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:30.981013 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:30.981260 (+ 247us) tablet.cc:699] Row locks acquired
1211 08:57:30.981273 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:30.981376 (+ 103us) write_op.cc:270] Start()
1211 08:57:30.981450 (+ 74us) write_op.cc:276] Timestamp: P: 1765443450981546 usec, L: 0
1211 08:57:30.981464 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:30.981893 (+ 429us) log.cc:844] Serialized 907 byte log entry
1211 08:57:31.062988 (+ 81095us) op_driver.cc:464] REPLICATION: finished
1211 08:57:31.063191 (+ 203us) write_op.cc:301] APPLY: starting
1211 08:57:31.063351 (+ 160us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:31.063468 (+ 117us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:31.063562 (+ 94us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:31.063771 (+ 209us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:31.063920 (+ 149us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:31.063958 (+ 38us) write_op.cc:312] APPLY: finished
1211 08:57:31.064112 (+ 154us) log.cc:844] Serialized 29 byte log entry
1211 08:57:31.064742 (+ 630us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:31.064823 (+ 81us) write_op.cc:454] Released schema lock
1211 08:57:31.064859 (+ 36us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:31.073859 (+ 9000us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:31.073884 (+ 25us) write_op.cc:454] Released schema lock
Related trace 'op':
1211 08:57:31.067469 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:31.067833 (+ 364us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:31.067851 (+ 18us) write_op.cc:435] Acquired schema lock
1211 08:57:31.067864 (+ 13us) tablet.cc:660] Decoding operations
1211 08:57:31.068084 (+ 220us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:31.068122 (+ 38us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:31.068145 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:31.068285 (+ 140us) tablet.cc:699] Row locks acquired
1211 08:57:31.068298 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:31.068394 (+ 96us) write_op.cc:270] Start()
1211 08:57:31.068464 (+ 70us) write_op.cc:276] Timestamp: P: 1765443451068395 usec, L: 0
1211 08:57:31.068479 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:57:31.068933 (+ 454us) log.cc:844] Serialized 356 byte log entry
1211 08:57:31.075324 (+ 6391us) op_driver.cc:464] REPLICATION: finished
1211 08:57:31.075863 (+ 539us) write_op.cc:301] APPLY: starting
1211 08:57:31.075928 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:31.076034 (+ 106us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:31.076133 (+ 99us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:31.076304 (+ 171us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:31.076475 (+ 171us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:31.076661 (+ 186us) write_op.cc:312] APPLY: finished
1211 08:57:31.076839 (+ 178us) log.cc:844] Serialized 23 byte log entry
1211 08:57:31.077376 (+ 537us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:31.077430 (+ 54us) write_op.cc:454] Released schema lock
1211 08:57:31.077463 (+ 33us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:31.077794 (+ 331us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:31.077818 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":11,"spinlock_wait_cycles":151715072,"child_traces":[["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":1298,"apply.run_wall_time_us":10631,"num_ops":2,"prepare.queue_time_us":8651,"prepare.run_cpu_time_us":1952,"prepare.run_wall_time_us":1951,"raft.queue_time_us":178,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":11,"replication_time_us":81426}],["op",{"apply.queue_time_us":393,"apply.run_cpu_time_us":1391,"apply.run_wall_time_us":1905,"mutex_wait_us":38,"num_ops":1,"prepare.queue_time_us":85,"prepare.run_cpu_time_us":1789,"prepare.run_wall_time_us":1816,"raft.queue_time_us":288,"raft.run_cpu_time_us":634,"raft.run_wall_time_us":636,"replication_time_us":6758,"spinlock_wait_cycles":8960}]]}
I20251211 08:57:32.038018 14904 ts_tablet_manager.cc:1929] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:32.038337 14904 log.cc:1199] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/c5191443773b415198687c55c4517692
I20251211 08:57:32.038944 14904 ts_tablet_manager.cc:1950] T c5191443773b415198687c55c4517692 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:32.039600 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 72) took 1075 ms. Trace:
I20251211 08:57:32.039911 14904 rpcz_store.cc:276] 1211 08:57:30.964575 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.965846 (+ 1271us) service_pool.cc:225] Handling call
1211 08:57:32.018661 (+1052815us) ts_tablet_manager.cc:1151] Deleting tablet c5191443773b415198687c55c4517692
1211 08:57:32.018697 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:32.032473 (+ 13776us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.037852 (+ 5379us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.039557 (+ 1705us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052284}
I20251211 08:57:32.041005 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet c5191443773b415198687c55c4517692 (table default.Table_4b521cb9e71f4e11a462261413eb85b4 [id=118053de49694cac881977b65f70b182]) successfully deleted
I20251211 08:57:32.041947 14904 tablet_replica.cc:333] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:32.042703 14904 raft_consensus.cc:2243] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:32.043443 14904 pending_rounds.cc:70] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
I20251211 08:57:32.043713 14904 pending_rounds.cc:77] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256379201581056 op_type: NO_OP noop_request { }
I20251211 08:57:32.044235 14904 raft_consensus.cc:2889] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:32.044605 14904 raft_consensus.cc:2272] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:32.047230 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f988855c0a7d4baf90dab6f27a1ed140 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:32.047829 14904 ts_tablet_manager.cc:1916] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:32.007442 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ccc5d49e4a09446bb4897be97093af29" } modify_external_catalogs: true
I20251211 08:57:32.052129 14718 ts_tablet_manager.cc:1929] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:32.052511 14718 log.cc:1199] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/45f3395a34cf4fdd8e1898b08d123c8a
I20251211 08:57:32.053220 14718 ts_tablet_manager.cc:1950] T 45f3395a34cf4fdd8e1898b08d123c8a P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
W20251211 08:57:32.063437 14230 leader_election.cc:343] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: Tablet not RUNNING: STOPPED
I20251211 08:57:32.063874 14230 leader_election.cc:304] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:32.064850 15035 raft_consensus.cc:2749] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:32.069432 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 74) took 1189 ms. Trace:
I20251211 08:57:32.069769 14718 rpcz_store.cc:276] 1211 08:57:30.879865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.885840 (+ 5975us) service_pool.cc:225] Handling call
1211 08:57:31.960261 (+1074421us) ts_tablet_manager.cc:1151] Deleting tablet 45f3395a34cf4fdd8e1898b08d123c8a
1211 08:57:31.960298 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:32.019977 (+ 59679us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.051901 (+ 31924us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.069371 (+ 17470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1073890}
I20251211 08:57:32.070751 14718 tablet_replica.cc:333] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:32.071174 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 45f3395a34cf4fdd8e1898b08d123c8a (table default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]) successfully deleted
I20251211 08:57:32.071460 14718 raft_consensus.cc:2243] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:32.071847 14718 raft_consensus.cc:2272] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:32.072723 14004 master_service.cc:954] ReplaceTablet: received request to replace tablet 25805a95349d49f58280a30e6876929d from {username='slave'} at 127.0.0.1:55614
W20251211 08:57:32.076045 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 40a1f12553934446bfe44f4ed41abf9a (table default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 40a1f12553934446bfe44f4ed41abf9a has no consensus state
I20251211 08:57:32.077608 15011 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:32.078274 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet c5191443773b415198687c55c4517692 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:32.084482 14722 tablet_bootstrap.cc:492] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:32.084932 14722 ts_tablet_manager.cc:1403] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.127s user 0.010s sys 0.000s
W20251211 08:57:31.861505 14010 rpcz_store.cc:269] 1211 08:57:29.817213 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.022807 (+205594us) service_pool.cc:225] Handling call
1211 08:57:30.090891 (+ 68084us) spinlock_profiling.cc:243] Waited 65.4 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:31.859749 (+1768858us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:30.643139 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:30.643526 (+ 387us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:30.643559 (+ 33us) write_op.cc:435] Acquired schema lock
1211 08:57:30.643573 (+ 14us) tablet.cc:660] Decoding operations
1211 08:57:30.643872 (+ 299us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:30.643912 (+ 40us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:30.643938 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:30.644214 (+ 276us) tablet.cc:699] Row locks acquired
1211 08:57:30.644229 (+ 15us) write_op.cc:260] PREPARE: finished
1211 08:57:30.644328 (+ 99us) write_op.cc:270] Start()
1211 08:57:30.644403 (+ 75us) write_op.cc:276] Timestamp: P: 1765443450644372 usec, L: 0
1211 08:57:30.644418 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:57:30.644860 (+ 442us) log.cc:844] Serialized 907 byte log entry
1211 08:57:30.725980 (+ 81120us) op_driver.cc:464] REPLICATION: finished
1211 08:57:30.727189 (+ 1209us) write_op.cc:301] APPLY: starting
1211 08:57:30.727261 (+ 72us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:30.727388 (+ 127us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:30.727503 (+ 115us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:30.727795 (+ 292us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:30.727965 (+ 170us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:30.728024 (+ 59us) write_op.cc:312] APPLY: finished
1211 08:57:30.728247 (+ 223us) log.cc:844] Serialized 29 byte log entry
1211 08:57:30.728396 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:30.728459 (+ 63us) write_op.cc:454] Released schema lock
1211 08:57:30.728490 (+ 31us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:30.728715 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:30.728747 (+ 32us) write_op.cc:454] Released schema lock
Related trace 'op':
1211 08:57:31.798025 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:31.798401 (+ 376us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:31.798437 (+ 36us) write_op.cc:435] Acquired schema lock
1211 08:57:31.798450 (+ 13us) tablet.cc:660] Decoding operations
1211 08:57:31.798686 (+ 236us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:31.798723 (+ 37us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:31.798744 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:31.798927 (+ 183us) tablet.cc:699] Row locks acquired
1211 08:57:31.798941 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:57:31.799063 (+ 122us) write_op.cc:270] Start()
1211 08:57:31.799136 (+ 73us) write_op.cc:276] Timestamp: P: 1765443451799017 usec, L: 0
1211 08:57:31.799149 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:31.799546 (+ 397us) log.cc:844] Serialized 356 byte log entry
1211 08:57:31.849790 (+ 50244us) op_driver.cc:464] REPLICATION: finished
1211 08:57:31.850561 (+ 771us) write_op.cc:301] APPLY: starting
1211 08:57:31.850616 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:31.850708 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:31.850832 (+ 124us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:31.850984 (+ 152us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:31.851123 (+ 139us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:31.851165 (+ 42us) write_op.cc:312] APPLY: finished
1211 08:57:31.851330 (+ 165us) log.cc:844] Serialized 23 byte log entry
1211 08:57:31.851457 (+ 127us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:31.851509 (+ 52us) write_op.cc:454] Released schema lock
1211 08:57:31.851539 (+ 30us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:31.851740 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:31.851773 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"spinlock_wait_cycles":150484352,"child_traces":[["op",{"apply.queue_time_us":1055,"apply.run_cpu_time_us":1500,"apply.run_wall_time_us":1498,"num_ops":2,"prepare.queue_time_us":485264,"prepare.run_cpu_time_us":1962,"prepare.run_wall_time_us":1961,"raft.queue_time_us":33893,"raft.run_cpu_time_us":724,"raft.run_wall_time_us":724,"replication_time_us":81472}],["op",{"apply.queue_time_us":600,"apply.run_cpu_time_us":1145,"apply.run_wall_time_us":1143,"num_ops":1,"prepare.queue_time_us":101,"prepare.run_cpu_time_us":1757,"prepare.run_wall_time_us":1774,"raft.queue_time_us":8903,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":16,"replication_time_us":50542}]]}
I20251211 08:57:32.087762 14722 raft_consensus.cc:359] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.090579 14722 raft_consensus.cc:385] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.091566 14722 raft_consensus.cc:740] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.093247 14722 consensus_queue.cc:260] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.102236 14600 raft_consensus.cc:2468] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:32.107939 14604 raft_consensus.cc:2468] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:32.109102 14230 leader_election.cc:304] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:32.110112 15035 raft_consensus.cc:2804] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:32.110507 15035 raft_consensus.cc:697] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:32.111186 15035 consensus_queue.cc:237] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.111807 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet 38655ee4996e41cc8f87b5248264613d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:32.119161 14714 tablet_bootstrap.cc:492] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:32.119541 14714 ts_tablet_manager.cc:1403] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.144s user 0.013s sys 0.015s
I20251211 08:57:32.121601 14714 raft_consensus.cc:359] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.122243 14714 raft_consensus.cc:385] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.122491 14714 raft_consensus.cc:740] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.131840 14905 ts_tablet_manager.cc:1929] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:32.132200 14905 log.cc:1199] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/e015d0fb9d6c4c80a510c05244b334e4
I20251211 08:57:32.132757 14722 ts_tablet_manager.cc:1434] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20251211 08:57:32.132871 14905 ts_tablet_manager.cc:1950] T e015d0fb9d6c4c80a510c05244b334e4 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:32.129448 14714 consensus_queue.cc:260] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.134941 14718 ts_tablet_manager.cc:1916] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:32.133206 15030 tablet_bootstrap.cc:492] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:32.134426 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 with delete_type TABLET_DATA_DELETED (replaced by tablet e6c4d4c99b3b496cab125e13153349cb) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:32.135468 15030 ts_tablet_manager.cc:1403] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.260s user 0.017s sys 0.012s
I20251211 08:57:32.148176 15030 raft_consensus.cc:359] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:32.149680 14008 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 32) took 898 ms (client timeout 1000 ms). Trace:
I20251211 08:57:32.155879 15030 raft_consensus.cc:385] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.158290 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 72) took 2117 ms. Trace:
I20251211 08:57:32.157673 15030 raft_consensus.cc:740] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.158689 14905 rpcz_store.cc:276] 1211 08:57:30.040551 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.041328 (+ 777us) service_pool.cc:225] Handling call
1211 08:57:31.759617 (+1718289us) ts_tablet_manager.cc:1151] Deleting tablet e015d0fb9d6c4c80a510c05244b334e4
1211 08:57:31.759663 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:31.948841 (+189178us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.131591 (+182750us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.158231 (+ 26640us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18293,"tablet-delete.queue_time_us":1717824}
I20251211 08:57:32.160033 14714 ts_tablet_manager.cc:1434] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20251211 08:57:32.144757 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet 45f3395a34cf4fdd8e1898b08d123c8a with delete_type TABLET_DATA_DELETED (replaced by tablet afe339c4cc95490683618e18a3da252b) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:32.145426 14722 tablet_bootstrap.cc:492] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:32.144783 14230 leader_election.cc:304] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:32.148936 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet d2c6080b3c1e4897bc1881e50f29f3a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:32.165024 14007 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 21) took 1053 ms (client timeout 1000 ms). Trace:
I20251211 08:57:32.138749 14904 ts_tablet_manager.cc:1929] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:32.166896 14904 log.cc:1199] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/00f79094ed8b4782a3a1488e41794f9b
I20251211 08:57:32.167619 14904 ts_tablet_manager.cc:1950] T 00f79094ed8b4782a3a1488e41794f9b P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:32.164878 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 45f3395a34cf4fdd8e1898b08d123c8a with delete_type TABLET_DATA_DELETED (replaced by tablet afe339c4cc95490683618e18a3da252b) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:32.168380 14905 tablet_replica.cc:333] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:32.172999 14718 ts_tablet_manager.cc:1929] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:32.173419 14718 log.cc:1199] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/00f79094ed8b4782a3a1488e41794f9b
I20251211 08:57:32.159207 15030 consensus_queue.cc:260] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.174230 14905 raft_consensus.cc:2243] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:32.174836 14905 raft_consensus.cc:2272] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:32.175283 15030 ts_tablet_manager.cc:1434] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20251211 08:57:32.176141 15030 tablet_bootstrap.cc:492] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:32.177731 14714 tablet_bootstrap.cc:492] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:32.168361 14722 tablet_bootstrap.cc:654] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:32.168614 15035 raft_consensus.cc:2804] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:32.166981 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet e015d0fb9d6c4c80a510c05244b334e4 (table default.Table_f647a2c75e7c46939a62573dea122fef [id=779be79e440a45d7ac4f6afb05bed936]) successfully deleted
I20251211 08:57:32.180122 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 00f79094ed8b4782a3a1488e41794f9b (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]) successfully deleted
W20251211 08:57:32.183254 14007 rpcz_store.cc:269] 1211 08:57:31.111753 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:31.111883 (+ 130us) service_pool.cc:225] Handling call
1211 08:57:32.164959 (+1053076us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":15}
I20251211 08:57:32.183797 15030 tablet_bootstrap.cc:654] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:32.185648 14714 tablet_bootstrap.cc:654] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:32.155720 14008 rpcz_store.cc:269] 1211 08:57:31.251519 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:31.292600 (+ 41081us) service_pool.cc:225] Handling call
1211 08:57:32.149602 (+857002us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:32.007988 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:32.008365 (+ 377us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:32.008393 (+ 28us) write_op.cc:435] Acquired schema lock
1211 08:57:32.008411 (+ 18us) tablet.cc:660] Decoding operations
1211 08:57:32.008648 (+ 237us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:32.008686 (+ 38us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:32.008707 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:32.008833 (+ 126us) tablet.cc:699] Row locks acquired
1211 08:57:32.008847 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:57:32.008943 (+ 96us) write_op.cc:270] Start()
1211 08:57:32.009024 (+ 81us) write_op.cc:276] Timestamp: P: 1765443452008906 usec, L: 0
1211 08:57:32.009043 (+ 19us) op_driver.cc:348] REPLICATION: starting
1211 08:57:32.009486 (+ 443us) log.cc:844] Serialized 530 byte log entry
1211 08:57:32.094172 (+ 84686us) op_driver.cc:464] REPLICATION: finished
1211 08:57:32.094550 (+ 378us) write_op.cc:301] APPLY: starting
1211 08:57:32.094599 (+ 49us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:32.094685 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:32.094761 (+ 76us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:32.094977 (+ 216us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:32.095170 (+ 193us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:32.095208 (+ 38us) write_op.cc:312] APPLY: finished
1211 08:57:32.095365 (+ 157us) log.cc:844] Serialized 23 byte log entry
1211 08:57:32.095488 (+ 123us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:32.095542 (+ 54us) write_op.cc:454] Released schema lock
1211 08:57:32.095573 (+ 31us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:32.095781 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:32.095803 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":250,"apply.run_cpu_time_us":1147,"apply.run_wall_time_us":1203,"num_ops":1,"prepare.queue_time_us":107,"prepare.run_cpu_time_us":1734,"prepare.run_wall_time_us":1733,"raft.queue_time_us":206,"raft.run_cpu_time_us":626,"raft.run_wall_time_us":626,"replication_time_us":85006}]]}
I20251211 08:57:32.187474 15038 raft_consensus.cc:493] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:32.187857 15038 raft_consensus.cc:515] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.189517 15038 leader_election.cc:290] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:32.190928 15035 raft_consensus.cc:697] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:32.191052 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "996c85729acd4418be1763b73cbf7d2d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:32.191924 15035 consensus_queue.cc:237] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.191578 14600 raft_consensus.cc:2468] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:32.195569 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "996c85729acd4418be1763b73cbf7d2d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:32.195586 14230 leader_election.cc:304] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:32.188575 14905 ts_tablet_manager.cc:1916] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:32.196799 15038 raft_consensus.cc:2804] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:32.197072 15038 raft_consensus.cc:493] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:32.197310 15038 raft_consensus.cc:3060] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:32.197530 14231 leader_election.cc:343] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:32.187299 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ba58b6955e7b4ab6ba45e12826d6f56d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:32.213932 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba58b6955e7b4ab6ba45e12826d6f56d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:32.215687 14904 tablet_replica.cc:333] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b: stopping tablet replica
W20251211 08:57:32.215961 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 60b23163906447c3877db631968f5623 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60b23163906447c3877db631968f5623 already in progress: creating tablet
I20251211 08:57:32.216449 14904 raft_consensus.cc:2243] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:32.216936 14904 raft_consensus.cc:2272] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:32.218698 14904 ts_tablet_manager.cc:1916] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:32.219508 14718 ts_tablet_manager.cc:1950] T 00f79094ed8b4782a3a1488e41794f9b P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:32.206357 15038 raft_consensus.cc:515] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.227600 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "996c85729acd4418be1763b73cbf7d2d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:32.228470 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 00f79094ed8b4782a3a1488e41794f9b (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]) successfully deleted
I20251211 08:57:32.229600 14718 tablet_replica.cc:333] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
W20251211 08:57:32.229475 14231 leader_election.cc:343] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:32.230846 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 60b23163906447c3877db631968f5623 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60b23163906447c3877db631968f5623 already in progress: creating tablet
I20251211 08:57:32.230356 14718 raft_consensus.cc:2243] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:32.232084 14718 raft_consensus.cc:2272] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:32.237821 14905 ts_tablet_manager.cc:1929] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:32.238394 14905 log.cc:1199] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/d2c6080b3c1e4897bc1881e50f29f3a0
I20251211 08:57:32.244267 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 25805a95349d49f58280a30e6876929d
I20251211 08:57:32.245376 14004 catalog_manager.cc:6724] ReplaceTablet: tablet 25805a95349d49f58280a30e6876929d deleted and replaced by tablet 41d6ef975a47489385791065bb8f9f46
I20251211 08:57:32.245482 14722 tablet_bootstrap.cc:492] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:32.245867 14722 ts_tablet_manager.cc:1403] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.012s
I20251211 08:57:32.246392 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 15f7bcd680064b9f8fe51aa05acce6fb
I20251211 08:57:32.247221 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 25805a95349d49f58280a30e6876929d with delete_type TABLET_DATA_DELETED (replaced by tablet 41d6ef975a47489385791065bb8f9f46) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:32.247484 14013 catalog_manager.cc:6724] ReplaceTablet: tablet 15f7bcd680064b9f8fe51aa05acce6fb deleted and replaced by tablet d4f826d079774152a3b66bfb81f990a8
I20251211 08:57:32.250312 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 15f7bcd680064b9f8fe51aa05acce6fb with delete_type TABLET_DATA_DELETED (replaced by tablet d4f826d079774152a3b66bfb81f990a8) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:32.251214 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 25805a95349d49f58280a30e6876929d with delete_type TABLET_DATA_DELETED (replaced by tablet 41d6ef975a47489385791065bb8f9f46) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:32.251363 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet 15f7bcd680064b9f8fe51aa05acce6fb with delete_type TABLET_DATA_DELETED (replaced by tablet d4f826d079774152a3b66bfb81f990a8) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:32.252538 14714 tablet_bootstrap.cc:492] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:32.254184 14714 ts_tablet_manager.cc:1403] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.006s
I20251211 08:57:32.255395 14905 ts_tablet_manager.cc:1950] T d2c6080b3c1e4897bc1881e50f29f3a0 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:32.257248 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 73) took 2188 ms. Trace:
I20251211 08:57:32.257524 14905 rpcz_store.cc:276] 1211 08:57:30.068736 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.069768 (+ 1032us) service_pool.cc:225] Handling call
1211 08:57:32.166111 (+2096343us) ts_tablet_manager.cc:1151] Deleting tablet d2c6080b3c1e4897bc1881e50f29f3a0
1211 08:57:32.168271 (+ 2160us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:32.202913 (+ 34642us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.214541 (+ 11628us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.257199 (+ 42658us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2095833}
I20251211 08:57:32.258522 14905 tablet_replica.cc:333] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:32.258502 15013 tablet.cc:1722] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:32.259475 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet d2c6080b3c1e4897bc1881e50f29f3a0 (table default.Table_86772d5552714163a6a0ffc236521623 [id=a5f4dfd6520045919362ac16b16d4acb]) successfully deleted
I20251211 08:57:32.260967 15010 tablet.cc:1722] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:32.260980 14008 catalog_manager.cc:5654] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:32.263603 15011 tablet.cc:1722] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:32.264151 14714 raft_consensus.cc:359] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.264665 14714 raft_consensus.cc:385] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.264890 14714 raft_consensus.cc:740] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.265509 14714 consensus_queue.cc:260] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.268504 14714 ts_tablet_manager.cc:1434] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.014s user 0.000s sys 0.004s
I20251211 08:57:32.270099 14905 raft_consensus.cc:2243] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:32.270512 14905 raft_consensus.cc:2272] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:32.262670 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 426e26859b9c4966b8ec8ba3eab53a43 (table default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 2 got 0
I20251211 08:57:32.279194 14722 raft_consensus.cc:359] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.279760 14722 raft_consensus.cc:385] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.279982 14722 raft_consensus.cc:740] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.242694 15030 tablet_bootstrap.cc:492] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:32.249696 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 25805a95349d49f58280a30e6876929d with delete_type TABLET_DATA_DELETED (replaced by tablet 41d6ef975a47489385791065bb8f9f46) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:32.250129 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet 15f7bcd680064b9f8fe51aa05acce6fb with delete_type TABLET_DATA_DELETED (replaced by tablet d4f826d079774152a3b66bfb81f990a8) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:32.280908 15030 ts_tablet_manager.cc:1403] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.004s
I20251211 08:57:32.283365 15030 raft_consensus.cc:359] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.284055 15030 raft_consensus.cc:385] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.284344 15030 raft_consensus.cc:740] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.284976 15030 consensus_queue.cc:260] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.289933 14006 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet f7457aa49b1c41fb87f747e8d35dd828
I20251211 08:57:32.290911 14714 tablet_bootstrap.cc:492] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:32.292490 14718 ts_tablet_manager.cc:1916] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:32.296183 14905 ts_tablet_manager.cc:1916] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:32.300714 14714 tablet_bootstrap.cc:654] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:32.302045 14722 consensus_queue.cc:260] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.312054 14722 ts_tablet_manager.cc:1434] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.066s user 0.003s sys 0.003s
I20251211 08:57:32.317229 14714 tablet_bootstrap.cc:492] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:32.317584 14714 ts_tablet_manager.cc:1403] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.000s
I20251211 08:57:32.317317 14722 tablet_bootstrap.cc:492] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:32.319624 14714 raft_consensus.cc:359] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.320183 14714 raft_consensus.cc:385] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.320420 15030 ts_tablet_manager.cc:1434] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20251211 08:57:32.320626 14714 raft_consensus.cc:740] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.321331 15030 tablet_bootstrap.cc:492] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:32.321305 14714 consensus_queue.cc:260] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.323753 14714 ts_tablet_manager.cc:1434] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251211 08:57:32.329767 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "996c85729acd4418be1763b73cbf7d2d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:32.330322 14600 raft_consensus.cc:3060] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:32.331015 14714 tablet_bootstrap.cc:492] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:32.336385 14600 raft_consensus.cc:2468] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:32.337327 14230 leader_election.cc:304] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:32.338193 15035 raft_consensus.cc:2804] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:32.338593 15035 raft_consensus.cc:697] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:32.339365 15035 consensus_queue.cc:237] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.340184 14714 tablet_bootstrap.cc:654] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:32.344362 14722 tablet_bootstrap.cc:654] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:32.348482 15038 leader_election.cc:290] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:32.350839 14718 ts_tablet_manager.cc:1929] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:32.343014 15018 raft_consensus.cc:493] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:32.351575 15018 raft_consensus.cc:515] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.353489 15018 leader_election.cc:290] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:32.355136 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6de43c843ff845a0bd4eb98b6054dd87" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:32.356302 14544 leader_election.cc:343] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:32.341950 15030 tablet_bootstrap.cc:654] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:32.354183 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6de43c843ff845a0bd4eb98b6054dd87" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:32.360333 14542 leader_election.cc:343] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:32.360780 14904 ts_tablet_manager.cc:1929] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:32.360718 14542 leader_election.cc:304] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:32.361112 14904 log.cc:1199] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/1853070f81c34b52b8c74a7402e6dad3
I20251211 08:57:32.361128 14714 tablet_bootstrap.cc:492] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:32.361498 14714 ts_tablet_manager.cc:1403] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.000s
I20251211 08:57:32.361759 14904 ts_tablet_manager.cc:1950] T 1853070f81c34b52b8c74a7402e6dad3 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:32.361877 15018 raft_consensus.cc:2749] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:32.362473 14718 log.cc:1199] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/1853070f81c34b52b8c74a7402e6dad3
I20251211 08:57:32.363214 14718 ts_tablet_manager.cc:1950] T 1853070f81c34b52b8c74a7402e6dad3 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:32.363830 14714 raft_consensus.cc:359] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.365489 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 1853070f81c34b52b8c74a7402e6dad3 (table default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4]) successfully deleted
W20251211 08:57:32.367271 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6de43c843ff845a0bd4eb98b6054dd87 consensus state has no leader
I20251211 08:57:32.368748 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 1853070f81c34b52b8c74a7402e6dad3 (table default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4]) successfully deleted
I20251211 08:57:32.369458 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8dd853c2cd9f4a18816d6724773e96cd" } new_table_name: "default.Table_91bdd3847f834c4b8500113d994514a5" modify_external_catalogs: true
I20251211 08:57:32.370539 15030 tablet_bootstrap.cc:492] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:32.370909 15030 ts_tablet_manager.cc:1403] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.003s
I20251211 08:57:32.371318 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_39c89fc0d98a45ad831ac604f107a406"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:32.372778 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39c89fc0d98a45ad831ac604f107a406 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:32.367301 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4d0f3ca2b6c74400bc10c59a0f8f55f0" } modify_external_catalogs: true
I20251211 08:57:32.375821 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4d0f3ca2b6c74400bc10c59a0f8f55f0" } modify_external_catalogs: true
W20251211 08:57:32.379786 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 2cd6e4352cfc41e497dc58d158e24ff2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cd6e4352cfc41e497dc58d158e24ff2 already in progress: creating tablet
W20251211 08:57:32.381071 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 38655ee4996e41cc8f87b5248264613d because the tablet was not found. No further retry: Not found: Tablet not found: 38655ee4996e41cc8f87b5248264613d
W20251211 08:57:32.382206 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet fab1ee5cf4eb4e11a72f5ad1de8c3856 because the tablet was not found. No further retry: Not found: Tablet not found: fab1ee5cf4eb4e11a72f5ad1de8c3856
I20251211 08:57:32.385715 14714 raft_consensus.cc:385] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.386031 14714 raft_consensus.cc:740] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.386730 14714 consensus_queue.cc:260] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:32.387539 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 45f3395a34cf4fdd8e1898b08d123c8a because the tablet was not found. No further retry: Not found: Tablet not found: 45f3395a34cf4fdd8e1898b08d123c8a
I20251211 08:57:32.401096 15030 raft_consensus.cc:359] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.401758 15030 raft_consensus.cc:385] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.401825 14905 ts_tablet_manager.cc:1929] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:32.402163 14905 log.cc:1199] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/45f3395a34cf4fdd8e1898b08d123c8a
I20251211 08:57:32.402068 15030 raft_consensus.cc:740] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
W20251211 08:57:32.411772 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet d2c6080b3c1e4897bc1881e50f29f3a0 because the tablet was not found. No further retry: Not found: Tablet not found: d2c6080b3c1e4897bc1881e50f29f3a0
W20251211 08:57:32.413259 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 25805a95349d49f58280a30e6876929d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25805a95349d49f58280a30e6876929d already in progress: creating tablet
I20251211 08:57:32.413964 14904 tablet_replica.cc:333] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:32.414664 14904 raft_consensus.cc:2243] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:32.415082 14904 raft_consensus.cc:2272] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:32.417135 14904 ts_tablet_manager.cc:1916] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:32.417526 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb5a6e533e1f4da7b03a800d029f712d consensus state has no leader
I20251211 08:57:32.418072 14013 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 965d4f8490444e2994ab86bd028b65e9
I20251211 08:57:32.420743 14722 tablet_bootstrap.cc:492] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:32.421070 14722 ts_tablet_manager.cc:1403] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.104s user 0.005s sys 0.007s
I20251211 08:57:32.423189 14722 raft_consensus.cc:359] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.423781 14722 raft_consensus.cc:385] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.424000 14722 raft_consensus.cc:740] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.424588 15030 consensus_queue.cc:260] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.429049 15030 ts_tablet_manager.cc:1434] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.058s user 0.005s sys 0.001s
I20251211 08:57:32.429569 14722 consensus_queue.cc:260] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.431063 14905 ts_tablet_manager.cc:1950] T 45f3395a34cf4fdd8e1898b08d123c8a P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:32.445649 14938 raft_consensus.cc:493] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:32.446049 14938 raft_consensus.cc:515] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.423137 14718 tablet_replica.cc:333] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:32.447326 14718 raft_consensus.cc:2243] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:32.447633 14938 leader_election.cc:290] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:32.448174 14718 pending_rounds.cc:70] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:32.448343 14718 pending_rounds.cc:77] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256371846377472 op_type: NO_OP noop_request { }
I20251211 08:57:32.448657 14718 raft_consensus.cc:2889] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:32.448709 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12379255548f49dfad17f3840a0c86bb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:32.448925 14718 raft_consensus.cc:2272] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:32.449311 14600 raft_consensus.cc:2393] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
I20251211 08:57:32.450454 14718 ts_tablet_manager.cc:1916] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:32.452096 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 74) took 1574 ms. Trace:
I20251211 08:57:32.452404 14905 rpcz_store.cc:276] 1211 08:57:30.877467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:30.877640 (+ 173us) service_pool.cc:225] Handling call
1211 08:57:32.258374 (+1380734us) ts_tablet_manager.cc:1151] Deleting tablet 45f3395a34cf4fdd8e1898b08d123c8a
1211 08:57:32.258413 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:32.362241 (+103828us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.401616 (+ 39375us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.452044 (+ 50428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380122}
I20251211 08:57:32.453497 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 45f3395a34cf4fdd8e1898b08d123c8a (table default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]) successfully deleted
I20251211 08:57:32.454365 14905 tablet_replica.cc:333] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:32.454614 14714 ts_tablet_manager.cc:1434] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.093s user 0.006s sys 0.000s
I20251211 08:57:32.455210 14905 raft_consensus.cc:2243] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:32.455632 14714 tablet_bootstrap.cc:492] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:32.455623 14905 raft_consensus.cc:2272] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:32.460186 14905 ts_tablet_manager.cc:1916] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:32.474720 14904 ts_tablet_manager.cc:1929] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:32.474994 14714 tablet_bootstrap.cc:654] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:32.475087 14904 log.cc:1199] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/15f7bcd680064b9f8fe51aa05acce6fb
I20251211 08:57:32.475661 14904 ts_tablet_manager.cc:1950] T 15f7bcd680064b9f8fe51aa05acce6fb P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:32.493438 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_adee98d887af4b9bb8d59ea922bd36f9" } modify_external_catalogs: true
I20251211 08:57:32.493907 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_adee98d887af4b9bb8d59ea922bd36f9" } modify_external_catalogs: true
I20251211 08:57:32.503382 14423 tablet_service.cc:1505] Processing CreateTablet for tablet a4e307e4d1994af0b99a88b3b06b73a4 (DEFAULT_TABLE table=default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.505999 14429 tablet_service.cc:1505] Processing CreateTablet for tablet c55408f39a414e2caf1db48e6f29eca7 (DEFAULT_TABLE table=default.Table_2a5d1e8d612c4fc5954ea3efc00eb00b [id=ad68b61e866c40f28e5dbd143ccfcb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.508147 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 363fe7a2e1ac4fb4a1ec97dc2354d71a (DEFAULT_TABLE table=default.Table_8dcae63f2e7547d290798fc98942d775 [id=5466128e36d8465b924c370f90739697]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.510433 14422 tablet_service.cc:1505] Processing CreateTablet for tablet caf537c461c04a81b66f268ee98dc64e (DEFAULT_TABLE table=default.Table_ac11e9583ef64c1bbae0b2940213b819 [id=8c183fcd39ae4bbeb0a05bfdd42e755e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.513548 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 383716a1f1f847ccbea542eca04790c9 (DEFAULT_TABLE table=default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.516026 14426 tablet_service.cc:1505] Processing CreateTablet for tablet afe339c4cc95490683618e18a3da252b (DEFAULT_TABLE table=default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.519642 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12379255548f49dfad17f3840a0c86bb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:32.520908 14428 tablet_service.cc:1505] Processing CreateTablet for tablet e6c4d4c99b3b496cab125e13153349cb (DEFAULT_TABLE table=default.Table_1ab8b3b3713f428293902db0aa8cedb6 [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.523684 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 15f7bcd680064b9f8fe51aa05acce6fb (table default.Table_d1d406b278304b5ca6e6cc3b63e7fd36 [id=4208ab42111742f38028caeb7913a1c7]) successfully deleted
I20251211 08:57:32.524333 15038 raft_consensus.cc:493] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:32.525131 14385 leader_election.cc:304] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:32.526559 14938 raft_consensus.cc:3060] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:32.528040 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 40a1f12553934446bfe44f4ed41abf9a (DEFAULT_TABLE table=default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.529726 14421 tablet_service.cc:1505] Processing CreateTablet for tablet 3c5315f708c04322919c714dc915a483 (DEFAULT_TABLE table=default.Table_d12516554d1547419ff004d016cb9a3a [id=1078a3f3dca34fbe8d48d184f434c7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.532250 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 14fc87cb76024b5b8227dfe9939e1c89 (DEFAULT_TABLE table=default.Table_7422c10fa9f240fe82576a2917407bef [id=6e2d344612214ee382863680435345b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.460312 15030 tablet_bootstrap.cc:492] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:32.466897 15018 raft_consensus.cc:493] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:32.500527 14587 tablet_service.cc:1505] Processing CreateTablet for tablet a4e307e4d1994af0b99a88b3b06b73a4 (DEFAULT_TABLE table=default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.536337 15018 raft_consensus.cc:515] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.536793 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e307e4d1994af0b99a88b3b06b73a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.538034 15018 leader_election.cc:290] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:32.507863 14584 tablet_service.cc:1505] Processing CreateTablet for tablet c55408f39a414e2caf1db48e6f29eca7 (DEFAULT_TABLE table=default.Table_2a5d1e8d612c4fc5954ea3efc00eb00b [id=ad68b61e866c40f28e5dbd143ccfcb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.539803 14938 raft_consensus.cc:2749] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:32.540745 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet a9524aa1503549bd820918d1c025aa2e from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:32.531252 14922 raft_consensus.cc:493] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:32.545790 14922 raft_consensus.cc:515] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.547026 14006 master_service.cc:954] ReplaceTablet: received request to replace tablet 426e26859b9c4966b8ec8ba3eab53a43 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:32.547442 14922 leader_election.cc:290] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:32.547467 14722 ts_tablet_manager.cc:1434] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.126s user 0.004s sys 0.011s
I20251211 08:57:32.549162 14722 tablet_bootstrap.cc:492] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:32.550990 14931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:57:32.551851 14931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20251211 08:57:32.552255 15038 raft_consensus.cc:515] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.554217 15038 leader_election.cc:290] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:32.555552 14275 tablet_service.cc:1505] Processing CreateTablet for tablet a4e307e4d1994af0b99a88b3b06b73a4 (DEFAULT_TABLE table=default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.556432 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e307e4d1994af0b99a88b3b06b73a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.509974 14581 tablet_service.cc:1505] Processing CreateTablet for tablet 363fe7a2e1ac4fb4a1ec97dc2354d71a (DEFAULT_TABLE table=default.Table_8dcae63f2e7547d290798fc98942d775 [id=5466128e36d8465b924c370f90739697]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.512281 14582 tablet_service.cc:1505] Processing CreateTablet for tablet caf537c461c04a81b66f268ee98dc64e (DEFAULT_TABLE table=default.Table_ac11e9583ef64c1bbae0b2940213b819 [id=8c183fcd39ae4bbeb0a05bfdd42e755e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.514551 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 383716a1f1f847ccbea542eca04790c9 (DEFAULT_TABLE table=default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.518627 14586 tablet_service.cc:1505] Processing CreateTablet for tablet afe339c4cc95490683618e18a3da252b (DEFAULT_TABLE table=default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.522997 14585 tablet_service.cc:1505] Processing CreateTablet for tablet e6c4d4c99b3b496cab125e13153349cb (DEFAULT_TABLE table=default.Table_1ab8b3b3713f428293902db0aa8cedb6 [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.533061 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 40a1f12553934446bfe44f4ed41abf9a (DEFAULT_TABLE table=default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.534562 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 3c5315f708c04322919c714dc915a483 (DEFAULT_TABLE table=default.Table_d12516554d1547419ff004d016cb9a3a [id=1078a3f3dca34fbe8d48d184f434c7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.568665 15030 tablet_bootstrap.cc:654] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:32.573938 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9168007ca0584954a734e2c8f2e73f70" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:32.579463 14931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:57:32.580324 14544 leader_election.cc:343] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:32.583118 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c55408f39a414e2caf1db48e6f29eca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.582489 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet a9524aa1503549bd820918d1c025aa2e
I20251211 08:57:32.584785 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 593df63696a04f85b00c301d93748e80 (table default.Table_f73c2258efaf4858800af6ac28f6d3a1 [id=ab93470ba8f54070be718c240d49aa13]). Expected version 3 got 2
I20251211 08:57:32.585500 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet a0f9c321e673433498a9059dd2e2efe6 (table default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5]). Expected version 1 got 0
I20251211 08:57:32.586370 14013 catalog_manager.cc:6724] ReplaceTablet: tablet a9524aa1503549bd820918d1c025aa2e deleted and replaced by tablet db302282bc3d4d8cb56d1cbb2f73c4fe
I20251211 08:57:32.587234 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25805a95349d49f58280a30e6876929d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:32.587707 14600 raft_consensus.cc:2468] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
W20251211 08:57:32.589980 14230 leader_election.cc:343] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: d235bade012540e1bb1c0dd871eff1bb
I20251211 08:57:32.590579 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a1f12553934446bfe44f4ed41abf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.590607 14230 leader_election.cc:304] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:32.591703 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf537c461c04a81b66f268ee98dc64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.594748 14579 tablet_service.cc:1505] Processing CreateTablet for tablet 14fc87cb76024b5b8227dfe9939e1c89 (DEFAULT_TABLE table=default.Table_7422c10fa9f240fe82576a2917407bef [id=6e2d344612214ee382863680435345b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.595235 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afe339c4cc95490683618e18a3da252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.595239 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 1976d584cc0a450ab809ead258a251e2 (DEFAULT_TABLE table=default.Table_81eee4b23e9045848867df8a0f511ae5 [id=72d8518a3dbf4dfdb28b1f422e44d414]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.595835 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fc87cb76024b5b8227dfe9939e1c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.596319 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1976d584cc0a450ab809ead258a251e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.596844 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5315f708c04322919c714dc915a483. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.598977 15038 raft_consensus.cc:2804] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:32.599269 15038 raft_consensus.cc:493] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:32.599891 15038 raft_consensus.cc:3060] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:32.602196 14277 tablet_service.cc:1505] Processing CreateTablet for tablet c55408f39a414e2caf1db48e6f29eca7 (DEFAULT_TABLE table=default.Table_2a5d1e8d612c4fc5954ea3efc00eb00b [id=ad68b61e866c40f28e5dbd143ccfcb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.602481 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 363fe7a2e1ac4fb4a1ec97dc2354d71a (DEFAULT_TABLE table=default.Table_8dcae63f2e7547d290798fc98942d775 [id=5466128e36d8465b924c370f90739697]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.554553 14419 tablet_service.cc:1505] Processing CreateTablet for tablet 1976d584cc0a450ab809ead258a251e2 (DEFAULT_TABLE table=default.Table_81eee4b23e9045848867df8a0f511ae5 [id=72d8518a3dbf4dfdb28b1f422e44d414]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.603103 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c55408f39a414e2caf1db48e6f29eca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.563637 14418 tablet_service.cc:1505] Processing CreateTablet for tablet 81cfb3e0baef45ddab4a77a4e0790085 (DEFAULT_TABLE table=default.Table_02e79cde30df4a2d8e81657161c2a56e [id=8ac1575fb83e405493d250160ca6ac24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.573203 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9168007ca0584954a734e2c8f2e73f70" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:32.604290 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_cf5e7d3231db4b53b9ba619336f5bb19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:32.605049 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363fe7a2e1ac4fb4a1ec97dc2354d71a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:32.605898 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf5e7d3231db4b53b9ba619336f5bb19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:32.605722 15038 raft_consensus.cc:515] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.607342 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_93954115dc5d4d2283faa424e83608b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:32.608566 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93954115dc5d4d2283faa424e83608b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:32.611510 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f988855c0a7d4baf90dab6f27a1ed140" } new_table_name: "default.Table_ccb3af786cad4564b032d78520da1cc3" modify_external_catalogs: true
I20251211 08:57:32.576252 14417 tablet_service.cc:1505] Processing CreateTablet for tablet b8d753358f20476ba23d7d7b72e26e74 (DEFAULT_TABLE table=default.Table_6cbe43becdc846bcab4b1233f65cda0c [id=8a6e51e5adbf4ca39770244bfdc843bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.586145 14416 tablet_service.cc:1552] Processing DeleteTablet for tablet a9524aa1503549bd820918d1c025aa2e with delete_type TABLET_DATA_DELETED (replaced by tablet db302282bc3d4d8cb56d1cbb2f73c4fe) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:32.615283 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e307e4d1994af0b99a88b3b06b73a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.615804 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afe339c4cc95490683618e18a3da252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.616360 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81cfb3e0baef45ddab4a77a4e0790085. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.616854 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1976d584cc0a450ab809ead258a251e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.617432 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5315f708c04322919c714dc915a483. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.618790 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a1f12553934446bfe44f4ed41abf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.619813 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fc87cb76024b5b8227dfe9939e1c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.620666 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d235bade012540e1bb1c0dd871eff1bb" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:32.621150 14448 raft_consensus.cc:2468] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:32.622550 14722 tablet_bootstrap.cc:654] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:32.623073 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363fe7a2e1ac4fb4a1ec97dc2354d71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.624300 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 383716a1f1f847ccbea542eca04790c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.624131 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25805a95349d49f58280a30e6876929d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:32.625293 14231 leader_election.cc:304] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:32.625577 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c55408f39a414e2caf1db48e6f29eca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.626247 15035 raft_consensus.cc:2804] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:32.626525 15035 raft_consensus.cc:493] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:32.626821 15035 raft_consensus.cc:3060] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:32.626906 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf537c461c04a81b66f268ee98dc64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.628329 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6c4d4c99b3b496cab125e13153349cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.629812 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25805a95349d49f58280a30e6876929d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:32.633074 14417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d753358f20476ba23d7d7b72e26e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.640969 14938 raft_consensus.cc:493] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:32.641381 14938 raft_consensus.cc:515] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.643152 14938 leader_election.cc:290] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:32.647781 14575 tablet_service.cc:1552] Processing DeleteTablet for tablet a9524aa1503549bd820918d1c025aa2e with delete_type TABLET_DATA_DELETED (replaced by tablet db302282bc3d4d8cb56d1cbb2f73c4fe) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:32.651532 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_358b3598ddbe4cc2831ba06d3600ac09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:32.655016 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6c4d4c99b3b496cab125e13153349cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.659555 15035 raft_consensus.cc:515] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.661695 15035 leader_election.cc:290] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:32.664139 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 383716a1f1f847ccbea542eca04790c9 (DEFAULT_TABLE table=default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.664413 14274 tablet_service.cc:1505] Processing CreateTablet for tablet afe339c4cc95490683618e18a3da252b (DEFAULT_TABLE table=default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.665215 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 383716a1f1f847ccbea542eca04790c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.665802 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afe339c4cc95490683618e18a3da252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.666602 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 383716a1f1f847ccbea542eca04790c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.671751 14278 tablet_service.cc:1505] Processing CreateTablet for tablet e6c4d4c99b3b496cab125e13153349cb (DEFAULT_TABLE table=default.Table_1ab8b3b3713f428293902db0aa8cedb6 [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.672734 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6c4d4c99b3b496cab125e13153349cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.673293 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25805a95349d49f58280a30e6876929d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:32.673799 14600 raft_consensus.cc:3060] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:32.654731 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_358b3598ddbe4cc2831ba06d3600ac09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:32.688400 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 63221.7us
I20251211 08:57:32.692685 15038 leader_election.cc:290] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:32.692816 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 40a1f12553934446bfe44f4ed41abf9a (DEFAULT_TABLE table=default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.693922 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a1f12553934446bfe44f4ed41abf9a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:32.695283 14231 leader_election.cc:343] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:32.695469 14272 tablet_service.cc:1505] Processing CreateTablet for tablet 3c5315f708c04322919c714dc915a483 (DEFAULT_TABLE table=default.Table_d12516554d1547419ff004d016cb9a3a [id=1078a3f3dca34fbe8d48d184f434c7b6]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:32.695816 14231 leader_election.cc:343] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:32.696416 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5315f708c04322919c714dc915a483. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.701969 14277 tablet_service.cc:1505] Processing CreateTablet for tablet caf537c461c04a81b66f268ee98dc64e (DEFAULT_TABLE table=default.Table_ac11e9583ef64c1bbae0b2940213b819 [id=8c183fcd39ae4bbeb0a05bfdd42e755e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.703269 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf537c461c04a81b66f268ee98dc64e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:32.659297 14542 leader_election.cc:343] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:32.709445 14271 tablet_service.cc:1505] Processing CreateTablet for tablet 14fc87cb76024b5b8227dfe9939e1c89 (DEFAULT_TABLE table=default.Table_7422c10fa9f240fe82576a2917407bef [id=6e2d344612214ee382863680435345b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.719158 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fc87cb76024b5b8227dfe9939e1c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.713593 14576 tablet_service.cc:1505] Processing CreateTablet for tablet b8d753358f20476ba23d7d7b72e26e74 (DEFAULT_TABLE table=default.Table_6cbe43becdc846bcab4b1233f65cda0c [id=8a6e51e5adbf4ca39770244bfdc843bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.718994 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363fe7a2e1ac4fb4a1ec97dc2354d71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.660490 14577 tablet_service.cc:1505] Processing CreateTablet for tablet 81cfb3e0baef45ddab4a77a4e0790085 (DEFAULT_TABLE table=default.Table_02e79cde30df4a2d8e81657161c2a56e [id=8ac1575fb83e405493d250160ca6ac24]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:32.721609 14230 leader_election.cc:343] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: d235bade012540e1bb1c0dd871eff1bb
I20251211 08:57:32.721995 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e015d0fb9d6c4c80a510c05244b334e4" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:32.722620 14297 raft_consensus.cc:2468] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:32.725157 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d235bade012540e1bb1c0dd871eff1bb" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:32.725807 14446 raft_consensus.cc:3060] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:32.742203 14011 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4550b341c9cc4c82b83b4daf817232bd" } new_table_name: "default.Table_0c2ec37766b24f539c7ff1e0aa4ac9c0" modify_external_catalogs: true
W20251211 08:57:32.754878 14382 leader_election.cc:343] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: e015d0fb9d6c4c80a510c05244b334e4
I20251211 08:57:32.756147 14718 ts_tablet_manager.cc:1929] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:32.756510 14718 log.cc:1199] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/7b60d99c2b4f442da4a4d9c5190f5797
I20251211 08:57:32.757266 14718 ts_tablet_manager.cc:1950] T 7b60d99c2b4f442da4a4d9c5190f5797 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:32.759037 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 88) took 1020 ms. Trace:
I20251211 08:57:32.771800 14268 tablet_service.cc:1505] Processing CreateTablet for tablet 81cfb3e0baef45ddab4a77a4e0790085 (DEFAULT_TABLE table=default.Table_02e79cde30df4a2d8e81657161c2a56e [id=8ac1575fb83e405493d250160ca6ac24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.773078 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81cfb3e0baef45ddab4a77a4e0790085. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.781214 14269 tablet_service.cc:1505] Processing CreateTablet for tablet 1976d584cc0a450ab809ead258a251e2 (DEFAULT_TABLE table=default.Table_81eee4b23e9045848867df8a0f511ae5 [id=72d8518a3dbf4dfdb28b1f422e44d414]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.782323 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1976d584cc0a450ab809ead258a251e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:32.782586 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.62:39015, user_credentials={real_user=slave}} blocked reactor thread for 88381.1us
I20251211 08:57:32.773674 14714 tablet_bootstrap.cc:492] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:32.789929 14714 ts_tablet_manager.cc:1403] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.334s user 0.010s sys 0.030s
I20251211 08:57:32.791867 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 426e26859b9c4966b8ec8ba3eab53a43
I20251211 08:57:32.776958 14446 raft_consensus.cc:2468] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:32.793221 14006 catalog_manager.cc:6724] ReplaceTablet: tablet 426e26859b9c4966b8ec8ba3eab53a43 deleted and replaced by tablet e15483f539f24f1ba1b442663d66fd45
I20251211 08:57:32.793282 14714 raft_consensus.cc:359] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.793836 14714 raft_consensus.cc:385] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.794070 14714 raft_consensus.cc:740] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.794874 14718 rpcz_store.cc:276] 1211 08:57:31.738213 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:31.738341 (+ 128us) service_pool.cc:225] Handling call
1211 08:57:32.423002 (+684661us) ts_tablet_manager.cc:1151] Deleting tablet 7b60d99c2b4f442da4a4d9c5190f5797
1211 08:57:32.423043 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:32.615228 (+192185us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.755954 (+140726us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.758987 (+ 3033us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":682555}
I20251211 08:57:32.794574 14714 consensus_queue.cc:260] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.800707 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 7b60d99c2b4f442da4a4d9c5190f5797 (table default.Table_cbeef8d5f95540b9a51b10a0a18a2dce [id=71569e18b5d347598f45f7e38c073515]) successfully deleted
W20251211 08:57:32.801348 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 53fc34c41afe44678b8d6dcf79a31f8b because the tablet was not found. No further retry: Not found: Tablet not found: 53fc34c41afe44678b8d6dcf79a31f8b
I20251211 08:57:32.801106 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet 426e26859b9c4966b8ec8ba3eab53a43 with delete_type TABLET_DATA_DELETED (replaced by tablet e15483f539f24f1ba1b442663d66fd45) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:32.801970 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 2cd6e4352cfc41e497dc58d158e24ff2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cd6e4352cfc41e497dc58d158e24ff2 already in progress: creating tablet
I20251211 08:57:32.804044 14385 leader_election.cc:304] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:32.806727 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 292b641df22f478d8849ff6148ecf951
I20251211 08:57:32.808313 14008 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_f73c2258efaf4858800af6ac28f6d3a1 [id=ab93470ba8f54070be718c240d49aa13] alter complete (version 3)
I20251211 08:57:32.808627 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_c15367b621c144348ec8511d2947fec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:32.810043 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c15367b621c144348ec8511d2947fec6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:32.807510 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_28a52518794d42d49dac3912b531c37c" } modify_external_catalogs: true
I20251211 08:57:32.812803 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_28a52518794d42d49dac3912b531c37c" } modify_external_catalogs: true
I20251211 08:57:32.809944 14938 raft_consensus.cc:2804] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:32.816740 14938 raft_consensus.cc:493] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:32.817008 14938 raft_consensus.cc:3060] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:32.820053 13999 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 -> Peer 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015): Couldn't send request to peer 73702d54989f4959acfc4a6fdcede6ca. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20251211 08:57:32.823032 14938 raft_consensus.cc:515] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.826838 14938 leader_election.cc:290] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:32.815104 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet 292b641df22f478d8849ff6148ecf951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:32 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:32.804765 14718 tablet_replica.cc:333] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:32.834254 14718 raft_consensus.cc:2243] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:32.834733 14718 raft_consensus.cc:2272] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:32.836277 14718 ts_tablet_manager.cc:1916] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:32.844620 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ad600554c0946e0bcb0da6a42be2ffd has no consensus state
I20251211 08:57:32.850925 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet a9524aa1503549bd820918d1c025aa2e with delete_type TABLET_DATA_DELETED (replaced by tablet db302282bc3d4d8cb56d1cbb2f73c4fe) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:32.851655 14904 tablet_replica.cc:333] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:32.854004 15018 raft_consensus.cc:493] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:32.854391 15018 raft_consensus.cc:515] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.855991 15018 leader_election.cc:290] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:32.857303 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet 292b641df22f478d8849ff6148ecf951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:32 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:32.775646 14542 leader_election.cc:304] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
W20251211 08:57:32.861800 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 203527us
I20251211 08:57:32.871798 15018 raft_consensus.cc:2749] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:32.878957 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 426e26859b9c4966b8ec8ba3eab53a43 with delete_type TABLET_DATA_DELETED (replaced by tablet e15483f539f24f1ba1b442663d66fd45) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:32.885221 14600 raft_consensus.cc:2468] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:32.886624 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81cfb3e0baef45ddab4a77a4e0790085. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.888073 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d753358f20476ba23d7d7b72e26e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.894201 15030 tablet_bootstrap.cc:492] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:32.828691 14714 ts_tablet_manager.cc:1434] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
W20251211 08:57:32.866288 14382 consensus_peers.cc:597] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38655ee4996e41cc8f87b5248264613d. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:32.918968 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 52819.6us
W20251211 08:57:32.919965 14382 leader_election.cc:343] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: e015d0fb9d6c4c80a510c05244b334e4
I20251211 08:57:32.871052 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cf1210733754f43885a7c0a350aa206" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:32.924784 14714 tablet_bootstrap.cc:492] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:32.933772 14714 tablet_bootstrap.cc:654] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:32.938143 14931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251211 08:57:32.938521 14931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:57:32.938840 14931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251211 08:57:32.939122 14931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251211 08:57:32.959954 15038 raft_consensus.cc:493] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:32.966704 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cf1210733754f43885a7c0a350aa206" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:32.970988 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 426e26859b9c4966b8ec8ba3eab53a43 with delete_type TABLET_DATA_DELETED (replaced by tablet e15483f539f24f1ba1b442663d66fd45) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:32.830814 14267 tablet_service.cc:1505] Processing CreateTablet for tablet b8d753358f20476ba23d7d7b72e26e74 (DEFAULT_TABLE table=default.Table_6cbe43becdc846bcab4b1233f65cda0c [id=8a6e51e5adbf4ca39770244bfdc843bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:32.972084 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d753358f20476ba23d7d7b72e26e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:32.977361 15030 ts_tablet_manager.cc:1403] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.517s user 0.007s sys 0.014s
I20251211 08:57:32.979807 15030 raft_consensus.cc:359] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.980436 15030 raft_consensus.cc:385] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.980696 15030 raft_consensus.cc:740] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.980885 14722 tablet_bootstrap.cc:492] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:32.981218 14722 ts_tablet_manager.cc:1403] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.432s user 0.028s sys 0.010s
I20251211 08:57:32.908010 14905 ts_tablet_manager.cc:1929] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:32.982314 14905 log.cc:1199] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/00f79094ed8b4782a3a1488e41794f9b
I20251211 08:57:32.982945 14905 ts_tablet_manager.cc:1950] T 00f79094ed8b4782a3a1488e41794f9b P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:32.984606 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 83) took 1329 ms. Trace:
I20251211 08:57:32.984870 14905 rpcz_store.cc:276] 1211 08:57:31.655348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:31.655576 (+ 228us) service_pool.cc:225] Handling call
1211 08:57:32.454246 (+798670us) ts_tablet_manager.cc:1151] Deleting tablet 00f79094ed8b4782a3a1488e41794f9b
1211 08:57:32.454279 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:32.650498 (+196219us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.907826 (+257328us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:32.984561 (+ 76735us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":775058}
W20251211 08:57:32.921643 14542 leader_election.cc:343] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:32.985872 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 64303.6us
I20251211 08:57:32.987568 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 00f79094ed8b4782a3a1488e41794f9b (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]) successfully deleted
I20251211 08:57:32.991600 14722 raft_consensus.cc:359] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.992173 14722 raft_consensus.cc:385] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:32.992414 14722 raft_consensus.cc:740] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:32.992893 14722 consensus_queue.cc:260] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:32.994248 14722 ts_tablet_manager.cc:1434] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20251211 08:57:32.994896 14230 leader_election.cc:304] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
W20251211 08:57:32.997620 14544 leader_election.cc:343] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:32.997977 14544 leader_election.cc:304] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:32.998935 14922 raft_consensus.cc:2804] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:33.000483 14008 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333] alter complete (version 1)
I20251211 08:57:33.004769 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 292b641df22f478d8849ff6148ecf951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:32 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:32.975121 14231 leader_election.cc:304] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:33.006300 14006 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet d211c92f70e84d3e8edf8ad0051422b2
I20251211 08:57:33.007678 15018 raft_consensus.cc:2749] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:33.008649 14938 raft_consensus.cc:493] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:33.009027 14938 raft_consensus.cc:515] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.012360 15030 consensus_queue.cc:260] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.014540 15030 ts_tablet_manager.cc:1434] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.037s user 0.004s sys 0.000s
I20251211 08:57:33.015375 15030 tablet_bootstrap.cc:492] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:32.972862 15038 raft_consensus.cc:515] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.016364 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_d8ac34f0cab345dd9b113ef1447ea361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:33.017699 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8ac34f0cab345dd9b113ef1447ea361 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:33.023150 15078 consensus_queue.cc:1048] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:33.024940 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f0b7cb6abf9b49fd8b0051bb2eb9e55e" } modify_external_catalogs: true
I20251211 08:57:33.025413 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f0b7cb6abf9b49fd8b0051bb2eb9e55e" } modify_external_catalogs: true
I20251211 08:57:33.026898 15038 leader_election.cc:290] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:33.027778 14922 raft_consensus.cc:697] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:33.028609 14922 consensus_queue.cc:237] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.030514 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e354416047f74d02a9455147ca8d6bf3" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:33.031030 14446 raft_consensus.cc:2468] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:33.036113 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e354416047f74d02a9455147ca8d6bf3" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:33.036625 14606 raft_consensus.cc:2468] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
W20251211 08:57:33.039433 14008 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 33) took 838 ms (client timeout 1000 ms). Trace:
I20251211 08:57:33.042985 14577 tablet_service.cc:1552] Processing DeleteTablet for tablet d2c6080b3c1e4897bc1881e50f29f3a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:33.036653 14010 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5] alter complete (version 1)
I20251211 08:57:33.012959 15035 raft_consensus.cc:2804] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:33.047504 14231 leader_election.cc:304] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:33.052306 15035 raft_consensus.cc:697] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:33.053253 14577 tablet_service.cc:1552] Processing DeleteTablet for tablet e015d0fb9d6c4c80a510c05244b334e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:33.053028 15035 consensus_queue.cc:237] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.055140 15078 raft_consensus.cc:2804] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:33.055416 15078 raft_consensus.cc:493] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:33.055665 15078 raft_consensus.cc:3060] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:33.059769 14938 leader_election.cc:290] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:33.060710 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b9b338a3f544ccb813247250b0f0eb7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:33.062587 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 7b9b338a3f544ccb813247250b0f0eb7
I20251211 08:57:33.063365 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet d235bade012540e1bb1c0dd871eff1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:28 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:33.064195 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet e015d0fb9d6c4c80a510c05244b334e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:33.065424 14577 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b9b338a3f544ccb813247250b0f0eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:33.066131 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b9b338a3f544ccb813247250b0f0eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.071913 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_face70f737874a76a60d1f4ea105380c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:33.075163 14905 tablet_replica.cc:333] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:57:33.075657 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_face70f737874a76a60d1f4ea105380c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:33.075870 14905 raft_consensus.cc:2243] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:33.077296 14722 tablet_bootstrap.cc:492] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:33.077628 13997 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6de43c843ff845a0bd4eb98b6054dd87 (table default.Table_3c7e457251e74eb7b8415bbcb6b6d7db [id=e5ffdeadfa904d85b68d0925da7e7a4c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6de43c843ff845a0bd4eb98b6054dd87 consensus state has no leader
I20251211 08:57:33.079192 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_fc4825b70b434d46a0295dc99cbbb4c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:33.080906 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc4825b70b434d46a0295dc99cbbb4c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:33.083461 14012 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 25) took 1036 ms (client timeout 1000 ms). Trace:
I20251211 08:57:33.083894 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_bab313cc214e46fb8e94ad2b35c2d938" } new_table_name: "default.Table_84c8a91dc6244ac0b8dd3a2986479da6" modify_external_catalogs: true
I20251211 08:57:33.085642 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b60d99c2b4f442da4a4d9c5190f5797 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.085950 14905 raft_consensus.cc:2272] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.087026 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet d235bade012540e1bb1c0dd871eff1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:28 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.087534 14905 ts_tablet_manager.cc:1916] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.088572 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 38655ee4996e41cc8f87b5248264613d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.076426 14577 tablet_service.cc:1552] Processing DeleteTablet for tablet 00f79094ed8b4782a3a1488e41794f9b with delete_type TABLET_DATA_DELETED (replaced by tablet 78c6a703e1e44f41931fe465def01bf6) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:33.089717 14012 rpcz_store.cc:269] 1211 08:57:32.046672 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.047718 (+ 1046us) service_pool.cc:225] Handling call
1211 08:57:33.083407 (+1035689us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23578,"reported_tablets":20}
I20251211 08:57:33.090516 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 1853070f81c34b52b8c74a7402e6dad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:30 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.093622 14274 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b9b338a3f544ccb813247250b0f0eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:33.094149 14385 consensus_peers.cc:597] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38655ee4996e41cc8f87b5248264613d. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:33.094779 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b9b338a3f544ccb813247250b0f0eb7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:33.039731 14008 rpcz_store.cc:269] 1211 08:57:32.200835 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.202053 (+ 1218us) service_pool.cc:225] Handling call
1211 08:57:33.039382 (+837329us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:32.369539 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:32.369937 (+ 398us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:32.369957 (+ 20us) write_op.cc:435] Acquired schema lock
1211 08:57:32.369976 (+ 19us) tablet.cc:660] Decoding operations
1211 08:57:32.370200 (+ 224us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:32.370231 (+ 31us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:32.370252 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:32.370368 (+ 116us) tablet.cc:699] Row locks acquired
1211 08:57:32.370378 (+ 10us) write_op.cc:260] PREPARE: finished
1211 08:57:32.370462 (+ 84us) write_op.cc:270] Start()
1211 08:57:32.370524 (+ 62us) write_op.cc:276] Timestamp: P: 1765443452370577 usec, L: 0
1211 08:57:32.370537 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:32.370935 (+ 398us) log.cc:844] Serialized 530 byte log entry
1211 08:57:32.390739 (+ 19804us) op_driver.cc:464] REPLICATION: finished
1211 08:57:32.392637 (+ 1898us) write_op.cc:301] APPLY: starting
1211 08:57:32.392684 (+ 47us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:32.392756 (+ 72us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:32.392853 (+ 97us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:32.392995 (+ 142us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:32.393102 (+ 107us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:32.393137 (+ 35us) write_op.cc:312] APPLY: finished
1211 08:57:32.393260 (+ 123us) log.cc:844] Serialized 23 byte log entry
1211 08:57:32.393359 (+ 99us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:32.393398 (+ 39us) write_op.cc:454] Released schema lock
1211 08:57:32.393422 (+ 24us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:32.393599 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:32.393622 (+ 23us) write_op.cc:454] Released schema lock
Related trace 'op':
1211 08:57:32.402909 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:32.404040 (+ 1131us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:32.404062 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:57:32.404073 (+ 11us) tablet.cc:660] Decoding operations
1211 08:57:32.404296 (+ 223us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:32.404329 (+ 33us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:32.404351 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:32.404457 (+ 106us) tablet.cc:699] Row locks acquired
1211 08:57:32.404470 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:32.404567 (+ 97us) write_op.cc:270] Start()
1211 08:57:32.404633 (+ 66us) write_op.cc:276] Timestamp: P: 1765443452404694 usec, L: 0
1211 08:57:32.404646 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:32.405058 (+ 412us) log.cc:844] Serialized 356 byte log entry
1211 08:57:32.437331 (+ 32273us) op_driver.cc:464] REPLICATION: finished
1211 08:57:32.440534 (+ 3203us) write_op.cc:301] APPLY: starting
1211 08:57:32.440580 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:32.440660 (+ 80us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:32.440739 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:32.440867 (+ 128us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:32.440990 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:32.441027 (+ 37us) write_op.cc:312] APPLY: finished
1211 08:57:32.441217 (+ 190us) log.cc:844] Serialized 23 byte log entry
1211 08:57:32.443856 (+ 2639us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:32.443935 (+ 79us) write_op.cc:454] Released schema lock
1211 08:57:32.443967 (+ 32us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:32.444161 (+ 194us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:32.444186 (+ 25us) write_op.cc:454] Released schema lock
Related trace 'op':
1211 08:57:32.838881 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:32.839474 (+ 593us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:32.839497 (+ 23us) write_op.cc:435] Acquired schema lock
1211 08:57:32.839512 (+ 15us) tablet.cc:660] Decoding operations
1211 08:57:32.839770 (+ 258us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:32.839806 (+ 36us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:32.839832 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:32.839964 (+ 132us) tablet.cc:699] Row locks acquired
1211 08:57:32.839977 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:32.840080 (+ 103us) write_op.cc:270] Start()
1211 08:57:32.840152 (+ 72us) write_op.cc:276] Timestamp: P: 1765443452840174 usec, L: 0
1211 08:57:32.840170 (+ 18us) op_driver.cc:348] REPLICATION: starting
1211 08:57:32.854035 (+ 13865us) log.cc:844] Serialized 356 byte log entry
1211 08:57:32.998098 (+144063us) op_driver.cc:464] REPLICATION: finished
1211 08:57:32.998306 (+ 208us) write_op.cc:301] APPLY: starting
1211 08:57:32.998361 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:32.998448 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:32.998530 (+ 82us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:32.998660 (+ 130us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:32.998788 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:32.998827 (+ 39us) write_op.cc:312] APPLY: finished
1211 08:57:32.998967 (+ 140us) log.cc:844] Serialized 23 byte log entry
1211 08:57:32.999092 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:32.999143 (+ 51us) write_op.cc:454] Released schema lock
1211 08:57:32.999169 (+ 26us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:32.999385 (+ 216us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:32.999406 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":1742,"apply.run_cpu_time_us":938,"apply.run_wall_time_us":937,"num_ops":1,"prepare.queue_time_us":87638,"prepare.run_cpu_time_us":1609,"prepare.run_wall_time_us":2877,"raft.queue_time_us":68,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":15,"replication_time_us":20132}],["op",{"apply.queue_time_us":3079,"apply.run_cpu_time_us":1109,"apply.run_wall_time_us":3604,"num_ops":1,"prepare.queue_time_us":1617,"prepare.run_cpu_time_us":1605,"prepare.run_wall_time_us":2378,"raft.queue_time_us":1204,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":16,"replication_time_us":32601}],["op",{"apply.queue_time_us":67,"apply.run_cpu_time_us":1042,"apply.run_wall_time_us":1042,"num_ops":1,"prepare.queue_time_us":26188,"prepare.run_cpu_time_us":1791,"prepare.run_wall_time_us":17091,"raft.queue_time_us":201,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":144447,"spinlock_wait_cycles":1920}]]}
I20251211 08:57:33.096000 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e015d0fb9d6c4c80a510c05244b334e4" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:33.096465 14296 raft_consensus.cc:3060] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:33.099974 14010 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 22) took 972 ms (client timeout 1000 ms). Trace:
W20251211 08:57:33.100312 14010 rpcz_store.cc:269] 1211 08:57:32.127638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.127754 (+ 116us) service_pool.cc:225] Handling call
1211 08:57:33.099921 (+972167us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:32.991213 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:32.991597 (+ 384us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:32.991619 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:57:32.991631 (+ 12us) tablet.cc:660] Decoding operations
1211 08:57:32.991852 (+ 221us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:32.991884 (+ 32us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:32.991905 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:32.992026 (+ 121us) tablet.cc:699] Row locks acquired
1211 08:57:32.992037 (+ 11us) write_op.cc:260] PREPARE: finished
1211 08:57:32.992131 (+ 94us) write_op.cc:270] Start()
1211 08:57:32.992200 (+ 69us) write_op.cc:276] Timestamp: P: 1765443452992289 usec, L: 0
1211 08:57:32.992213 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:32.992688 (+ 475us) log.cc:844] Serialized 356 byte log entry
1211 08:57:33.033905 (+ 41217us) op_driver.cc:464] REPLICATION: finished
1211 08:57:33.035134 (+ 1229us) write_op.cc:301] APPLY: starting
1211 08:57:33.035190 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:33.035279 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:33.035386 (+ 107us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:33.035532 (+ 146us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:33.035661 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:33.035705 (+ 44us) write_op.cc:312] APPLY: finished
1211 08:57:33.035859 (+ 154us) log.cc:844] Serialized 23 byte log entry
1211 08:57:33.035993 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:33.036050 (+ 57us) write_op.cc:454] Released schema lock
1211 08:57:33.036081 (+ 31us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:33.036283 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:33.036330 (+ 47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":13866,"reported_tablets":16,"spinlock_wait_cycles":640896,"child_traces":[["op",{"apply.queue_time_us":1093,"apply.run_cpu_time_us":1122,"apply.run_wall_time_us":1122,"num_ops":1,"prepare.queue_time_us":14526,"prepare.run_cpu_time_us":1691,"prepare.run_wall_time_us":1691,"raft.queue_time_us":204,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":11,"replication_time_us":41609}]]}
I20251211 08:57:33.106958 14904 raft_consensus.cc:2243] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:33.107736 14904 raft_consensus.cc:2272] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.109398 14904 ts_tablet_manager.cc:1916] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.111290 14385 leader_election.cc:304] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:33.112572 14938 raft_consensus.cc:2749] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:33.116765 15030 tablet_bootstrap.cc:654] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:33.118134 15018 raft_consensus.cc:493] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:33.118709 15018 raft_consensus.cc:515] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.121392 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccce5a446ac9460c9c9b7de72d875d95" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:33.122700 14542 leader_election.cc:343] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:33.123824 15018 leader_election.cc:290] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:33.130430 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 00f79094ed8b4782a3a1488e41794f9b with delete_type TABLET_DATA_DELETED (replaced by tablet 78c6a703e1e44f41931fe465def01bf6) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.137051 15030 tablet_bootstrap.cc:492] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:33.137439 15030 ts_tablet_manager.cc:1403] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.122s user 0.005s sys 0.004s
I20251211 08:57:33.139288 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccce5a446ac9460c9c9b7de72d875d95" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:33.139869 15030 raft_consensus.cc:359] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.140496 15030 raft_consensus.cc:385] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251211 08:57:33.140770 14544 leader_election.cc:343] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:33.141232 14544 leader_election.cc:304] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:33.142156 15018 raft_consensus.cc:2749] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:33.149209 14904 ts_tablet_manager.cc:1929] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:33.149560 14904 log.cc:1199] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/a9524aa1503549bd820918d1c025aa2e
I20251211 08:57:33.150247 14904 ts_tablet_manager.cc:1950] T a9524aa1503549bd820918d1c025aa2e P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:33.140770 15030 raft_consensus.cc:740] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:33.135183 14296 raft_consensus.cc:2468] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:33.154853 14385 leader_election.cc:304] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:33.155676 15083 raft_consensus.cc:2804] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:33.156095 15083 raft_consensus.cc:697] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:33.156852 15083 consensus_queue.cc:237] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.161398 14938 raft_consensus.cc:493] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:33.161938 14938 raft_consensus.cc:515] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.164276 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet a9524aa1503549bd820918d1c025aa2e (table default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]) successfully deleted
I20251211 08:57:33.166891 14904 tablet_replica.cc:333] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:33.167207 14938 leader_election.cc:290] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:33.167142 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "996c85729acd4418be1763b73cbf7d2d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:33.167991 14606 raft_consensus.cc:2393] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
I20251211 08:57:33.168283 14904 raft_consensus.cc:2243] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:33.169018 14904 raft_consensus.cc:2272] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.171419 15030 consensus_queue.cc:260] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.173614 15030 ts_tablet_manager.cc:1434] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.036s user 0.000s sys 0.005s
I20251211 08:57:33.174432 15030 tablet_bootstrap.cc:492] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:33.190585 14904 ts_tablet_manager.cc:1916] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.195969 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7b8ca0c73d454d97a2aee4c1d08adece" } new_table_name: "default.Table_58593e18b9634b2c8912ff1109e05a52" modify_external_catalogs: true
I20251211 08:57:33.215765 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_3360abe9961b4a2fa432c21fd0c577a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:33.217303 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3360abe9961b4a2fa432c21fd0c577a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:33.220366 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4c07a2c494ed47d29a486da1035027e3" } modify_external_catalogs: true
I20251211 08:57:33.220847 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4c07a2c494ed47d29a486da1035027e3" } modify_external_catalogs: true
I20251211 08:57:33.223081 14904 ts_tablet_manager.cc:1929] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:33.226962 14904 log.cc:1199] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/426e26859b9c4966b8ec8ba3eab53a43
I20251211 08:57:33.143885 15078 raft_consensus.cc:515] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.227842 14904 ts_tablet_manager.cc:1950] T 426e26859b9c4966b8ec8ba3eab53a43 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:57:33.229543 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42720f5f5bf445f2b72e513ad4a6d95f has no consensus state
I20251211 08:57:33.237407 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "996c85729acd4418be1763b73cbf7d2d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:33.238853 14385 leader_election.cc:304] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:33.239553 15083 raft_consensus.cc:3060] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:33.242841 15078 leader_election.cc:290] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:33.245836 15083 raft_consensus.cc:2749] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:33.251899 15030 tablet_bootstrap.cc:654] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:33.257174 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e354416047f74d02a9455147ca8d6bf3" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:33.257730 14606 raft_consensus.cc:3060] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:33.263796 14606 raft_consensus.cc:2468] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:33.271324 15099 tablet.cc:1722] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:33.276446 14714 tablet_bootstrap.cc:492] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:33.276870 14714 ts_tablet_manager.cc:1403] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.352s user 0.006s sys 0.004s
I20251211 08:57:33.276264 15100 tablet.cc:1722] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:33.279608 14718 ts_tablet_manager.cc:1929] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:33.280009 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ec0af779c90941db9c7bf85ccff7499f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:33.281507 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec0af779c90941db9c7bf85ccff7499f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:33.283749 14012 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_3360abe9961b4a2fa432c21fd0c577a0" } modify_external_catalogs: true
I20251211 08:57:33.284253 14012 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_3360abe9961b4a2fa432c21fd0c577a0" } modify_external_catalogs: true
I20251211 08:57:33.285210 14718 log.cc:1199] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/c5191443773b415198687c55c4517692
I20251211 08:57:33.285859 14718 ts_tablet_manager.cc:1950] T c5191443773b415198687c55c4517692 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:33.287735 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 92) took 1209 ms. Trace:
I20251211 08:57:33.288172 14230 leader_election.cc:304] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:33.288013 14718 rpcz_store.cc:276] 1211 08:57:32.078054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.078130 (+ 76us) service_pool.cc:225] Handling call
1211 08:57:32.804645 (+726515us) ts_tablet_manager.cc:1151] Deleting tablet c5191443773b415198687c55c4517692
1211 08:57:32.804679 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:33.184536 (+379857us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:33.279349 (+ 94813us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:33.287680 (+ 8331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":726010}
I20251211 08:57:33.290995 15102 tablet.cc:1722] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:33.298707 14922 raft_consensus.cc:2804] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:33.298722 15038 consensus_queue.cc:1048] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:33.300047 14922 raft_consensus.cc:697] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:33.300586 15038 consensus_queue.cc:1048] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:33.300781 14922 consensus_queue.cc:237] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.304302 15089 raft_consensus.cc:493] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:33.304757 15089 raft_consensus.cc:515] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.309015 15089 leader_election.cc:290] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:33.309669 14722 tablet_bootstrap.cc:654] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:33.311426 14449 raft_consensus.cc:3060] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:33.313594 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3303dc521d694be6b386a35da1172ede" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:33.314543 15101 tablet.cc:1722] T 559f9d428b5841708999725b5b859660 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:33.316711 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet c5191443773b415198687c55c4517692 (table default.Table_4b521cb9e71f4e11a462261413eb85b4 [id=118053de49694cac881977b65f70b182]) successfully deleted
I20251211 08:57:33.318207 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_5a97657fb7304e73bddac0c22b7ee787"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:33.319600 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a97657fb7304e73bddac0c22b7ee787 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:33.317941 14542 leader_election.cc:343] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:33.316884 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 95) took 1152 ms. Trace:
I20251211 08:57:33.326233 14718 rpcz_store.cc:276] 1211 08:57:32.164600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.164735 (+ 135us) service_pool.cc:225] Handling call
1211 08:57:33.316682 (+1151947us) ts_tablet_manager.cc:1151] Deleting tablet 45f3395a34cf4fdd8e1898b08d123c8a
1211 08:57:33.316716 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:33.316840 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148684}
I20251211 08:57:33.326591 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a939f5e4faf546148710852f96ded5d8" } new_table_name: "default.Table_2d7127602bd94e2dbc13d3cfae809a87" modify_external_catalogs: true
W20251211 08:57:33.328120 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 45f3395a34cf4fdd8e1898b08d123c8a because the tablet was not found. No further retry: Not found: Tablet not found: 45f3395a34cf4fdd8e1898b08d123c8a
I20251211 08:57:33.331061 14905 ts_tablet_manager.cc:1929] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:33.331884 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 96) took 1085 ms. Trace:
I20251211 08:57:33.338879 14718 rpcz_store.cc:276] 1211 08:57:32.246893 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.247079 (+ 186us) service_pool.cc:225] Handling call
1211 08:57:33.331582 (+1084503us) ts_tablet_manager.cc:1151] Deleting tablet 25805a95349d49f58280a30e6876929d
1211 08:57:33.331620 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:33.331834 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1083969}
I20251211 08:57:33.339680 14905 log.cc:1199] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/60b23163906447c3877db631968f5623
I20251211 08:57:33.340343 14905 ts_tablet_manager.cc:1950] T 60b23163906447c3877db631968f5623 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:57:33.340992 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 25805a95349d49f58280a30e6876929d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25805a95349d49f58280a30e6876929d already in progress: creating tablet
I20251211 08:57:33.341171 14904 tablet_replica.cc:333] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:33.341486 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 97) took 1091 ms. Trace:
I20251211 08:57:33.341889 14904 raft_consensus.cc:2243] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:33.342406 14904 raft_consensus.cc:2272] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.342447 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 426e26859b9c4966b8ec8ba3eab53a43 (table default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6]) successfully deleted
I20251211 08:57:33.344069 14904 ts_tablet_manager.cc:1916] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.343953 14718 rpcz_store.cc:276] 1211 08:57:32.249964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.250188 (+ 224us) service_pool.cc:225] Handling call
1211 08:57:33.341082 (+1090894us) ts_tablet_manager.cc:1151] Deleting tablet 15f7bcd680064b9f8fe51aa05acce6fb
1211 08:57:33.341118 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:33.341438 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090397}
I20251211 08:57:33.348624 14718 tablet_replica.cc:333] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
W20251211 08:57:33.348862 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 15f7bcd680064b9f8fe51aa05acce6fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15f7bcd680064b9f8fe51aa05acce6fb already in progress: creating tablet
I20251211 08:57:33.348830 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7c905b0fe6de4e57bb2c49d7bf87206a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:33.349392 15035 raft_consensus.cc:493] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:33.349490 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 84) took 1646 ms. Trace:
I20251211 08:57:33.348451 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e354416047f74d02a9455147ca8d6bf3" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:33.350071 15030 tablet_bootstrap.cc:492] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:33.349766 15035 raft_consensus.cc:515] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.350314 14448 raft_consensus.cc:3060] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:33.350335 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c905b0fe6de4e57bb2c49d7bf87206a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:33.350472 15030 ts_tablet_manager.cc:1403] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.176s user 0.010s sys 0.000s
I20251211 08:57:33.351469 15035 leader_election.cc:290] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:33.351963 15035 consensus_queue.cc:1048] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:33.356380 14448 raft_consensus.cc:2468] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:33.357520 14718 raft_consensus.cc:2243] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:33.358050 14718 raft_consensus.cc:2272] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.359889 14718 ts_tablet_manager.cc:1916] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.368829 15035 consensus_queue.cc:1048] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:33.371861 15030 raft_consensus.cc:359] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.372720 15030 raft_consensus.cc:385] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:33.373014 15030 raft_consensus.cc:740] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:33.374583 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60b23163906447c3877db631968f5623" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:33.349761 14905 rpcz_store.cc:276] 1211 08:57:31.712355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:31.715068 (+ 2713us) service_pool.cc:225] Handling call
1211 08:57:33.074993 (+1359925us) ts_tablet_manager.cc:1151] Deleting tablet 60b23163906447c3877db631968f5623
1211 08:57:33.075031 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:33.257133 (+182102us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:33.330862 (+ 73729us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:33.349437 (+ 18575us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1359427}
I20251211 08:57:33.376011 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 60b23163906447c3877db631968f5623 (table default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]) successfully deleted
I20251211 08:57:33.377183 14905 tablet_replica.cc:333] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:33.377628 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3303dc521d694be6b386a35da1172ede" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:33.380930 14231 leader_election.cc:343] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:33.382159 14905 raft_consensus.cc:2243] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251211 08:57:33.382593 14544 leader_election.cc:343] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:33.384765 14544 leader_election.cc:304] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:33.385733 15089 raft_consensus.cc:2749] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:33.386595 14905 raft_consensus.cc:2272] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.388331 14427 tablet_service.cc:1505] Processing CreateTablet for tablet e8a8437b8b464fb8ab02a9c552f81db3 (DEFAULT_TABLE table=default.Table_39c89fc0d98a45ad831ac604f107a406 [id=0944cc290d954cf4a33fec11acb53315]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.389462 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8a8437b8b464fb8ab02a9c552f81db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.394366 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 559f9d428b5841708999725b5b859660 (table default.Table_84c8a91dc6244ac0b8dd3a2986479da6 [id=3f3f000cd93e44df9f084873ba4b168b]). Expected version 1 got 0
I20251211 08:57:33.395020 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 1a2dcb9ac62e4aac885d638f4e2b072e (table default.Table_0c2ec37766b24f539c7ff1e0aa4ac9c0 [id=1673af33c2f846c7bf092303eff6e8ad]). Expected version 3 got 2
I20251211 08:57:33.401268 14010 catalog_manager.cc:5654] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:33.403193 14010 catalog_manager.cc:5654] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:33.401964 14905 ts_tablet_manager.cc:1916] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.415926 14012 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 90991b695259417996ba42d27525ff7f
I20251211 08:57:33.423034 15030 consensus_queue.cc:260] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.433878 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_94b52b113a9f45b69bff699f4d2bdd0f" } new_table_name: "default.Table_fa2830322a8a4f7b96cd19543b967b05" modify_external_catalogs: true
W20251211 08:57:33.434262 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 51974.4us
W20251211 08:57:33.434892 14230 leader_election.cc:343] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 60b23163906447c3877db631968f5623
I20251211 08:57:33.441426 15030 ts_tablet_manager.cc:1434] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.091s user 0.001s sys 0.005s
I20251211 08:57:33.447634 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet a4e307e4d1994af0b99a88b3b06b73a4 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:33.448348 14718 ts_tablet_manager.cc:1929] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:33.448681 14718 log.cc:1199] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/a9524aa1503549bd820918d1c025aa2e
I20251211 08:57:33.449422 14718 ts_tablet_manager.cc:1950] T a9524aa1503549bd820918d1c025aa2e P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:33.452898 14011 master_service.cc:954] ReplaceTablet: received request to replace tablet c181adf9db574ab39432b2f9b14fe4d7 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:33.460436 14267 tablet_service.cc:1505] Processing CreateTablet for tablet e8a8437b8b464fb8ab02a9c552f81db3 (DEFAULT_TABLE table=default.Table_39c89fc0d98a45ad831ac604f107a406 [id=0944cc290d954cf4a33fec11acb53315]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.461311 14584 tablet_service.cc:1505] Processing CreateTablet for tablet e8a8437b8b464fb8ab02a9c552f81db3 (DEFAULT_TABLE table=default.Table_39c89fc0d98a45ad831ac604f107a406 [id=0944cc290d954cf4a33fec11acb53315]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.462038 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 5daba165262e41af8d3481d3b4929092 (DEFAULT_TABLE table=default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.462328 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8a8437b8b464fb8ab02a9c552f81db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.462791 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8a8437b8b464fb8ab02a9c552f81db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.463006 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5daba165262e41af8d3481d3b4929092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.464303 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_10e56e6be8ad4d798a3d979bbb66ffc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:33.465559 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10e56e6be8ad4d798a3d979bbb66ffc3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:33.468140 14429 tablet_service.cc:1505] Processing CreateTablet for tablet 42720f5f5bf445f2b72e513ad4a6d95f (DEFAULT_TABLE table=default.Table_58593e18b9634b2c8912ff1109e05a52 [id=8bb6612dda374661a7d3a8a06bd230ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.468568 14424 tablet_service.cc:1505] Processing CreateTablet for tablet d4f826d079774152a3b66bfb81f990a8 (DEFAULT_TABLE table=default.Table_d1d406b278304b5ca6e6cc3b63e7fd36 [id=4208ab42111742f38028caeb7913a1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.468619 14577 tablet_service.cc:1505] Processing CreateTablet for tablet 5daba165262e41af8d3481d3b4929092 (DEFAULT_TABLE table=default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.468915 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42720f5f5bf445f2b72e513ad4a6d95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.469555 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f826d079774152a3b66bfb81f990a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.469566 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5daba165262e41af8d3481d3b4929092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.471494 14581 tablet_service.cc:1505] Processing CreateTablet for tablet 42720f5f5bf445f2b72e513ad4a6d95f (DEFAULT_TABLE table=default.Table_58593e18b9634b2c8912ff1109e05a52 [id=8bb6612dda374661a7d3a8a06bd230ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.472512 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42720f5f5bf445f2b72e513ad4a6d95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.477872 14269 tablet_service.cc:1505] Processing CreateTablet for tablet d4f826d079774152a3b66bfb81f990a8 (DEFAULT_TABLE table=default.Table_d1d406b278304b5ca6e6cc3b63e7fd36 [id=4208ab42111742f38028caeb7913a1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.478821 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f826d079774152a3b66bfb81f990a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.483304 15030 tablet_bootstrap.cc:492] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:33.483613 14268 tablet_service.cc:1505] Processing CreateTablet for tablet 42720f5f5bf445f2b72e513ad4a6d95f (DEFAULT_TABLE table=default.Table_58593e18b9634b2c8912ff1109e05a52 [id=8bb6612dda374661a7d3a8a06bd230ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.484557 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42720f5f5bf445f2b72e513ad4a6d95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.487023 14905 ts_tablet_manager.cc:1929] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:33.487332 14905 log.cc:1199] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/1853070f81c34b52b8c74a7402e6dad3
I20251211 08:57:33.488036 14905 ts_tablet_manager.cc:1950] T 1853070f81c34b52b8c74a7402e6dad3 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:33.489084 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet c5191443773b415198687c55c4517692 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:27 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.490603 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet e015d0fb9d6c4c80a510c05244b334e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.492604 14271 tablet_service.cc:1505] Processing CreateTablet for tablet 5daba165262e41af8d3481d3b4929092 (DEFAULT_TABLE table=default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.493501 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5daba165262e41af8d3481d3b4929092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.495756 14718 tablet_replica.cc:333] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:33.496454 14718 raft_consensus.cc:2243] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:33.497812 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet a9524aa1503549bd820918d1c025aa2e (table default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]) successfully deleted
I20251211 08:57:33.504253 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 85) took 1777 ms. Trace:
I20251211 08:57:33.504539 14905 rpcz_store.cc:276] 1211 08:57:31.727080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:31.727927 (+ 847us) service_pool.cc:225] Handling call
1211 08:57:33.377004 (+1649077us) ts_tablet_manager.cc:1151] Deleting tablet 1853070f81c34b52b8c74a7402e6dad3
1211 08:57:33.377064 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:33.451716 (+ 74652us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:33.486821 (+ 35105us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:33.504203 (+ 17382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1648488}
I20251211 08:57:33.505597 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 1853070f81c34b52b8c74a7402e6dad3 (table default.Table_eb1d2cda272c4981a164a7db6301ec2c [id=251c1a544d17439eb2a0a927454b44d4]) successfully deleted
I20251211 08:57:33.496917 14718 raft_consensus.cc:2272] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.507988 14718 ts_tablet_manager.cc:1916] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.435284 14230 leader_election.cc:304] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:33.514103 15078 consensus_queue.cc:1048] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:33.514844 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 57936.3us
W20251211 08:57:33.516417 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 81599.2us
I20251211 08:57:33.517580 14922 raft_consensus.cc:2749] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:33.523131 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 87) took 1662 ms. Trace:
I20251211 08:57:33.523473 14905 rpcz_store.cc:276] 1211 08:57:31.860953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:31.861523 (+ 570us) service_pool.cc:225] Handling call
1211 08:57:33.511845 (+1650322us) ts_tablet_manager.cc:1151] Deleting tablet 7d4f6727c4464cc9b7d68866c3969a0f
1211 08:57:33.511890 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:33.523078 (+ 11188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1649719}
I20251211 08:57:33.524564 15030 tablet_bootstrap.cc:654] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:33.524832 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 7d4f6727c4464cc9b7d68866c3969a0f because the tablet was not found. No further retry: Not found: Tablet not found: 7d4f6727c4464cc9b7d68866c3969a0f
I20251211 08:57:33.527298 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 88) took 1506 ms. Trace:
I20251211 08:57:33.527585 14905 rpcz_store.cc:276] 1211 08:57:32.020941 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.022904 (+ 1963us) service_pool.cc:225] Handling call
1211 08:57:33.526103 (+1503199us) ts_tablet_manager.cc:1151] Deleting tablet 2cd6e4352cfc41e497dc58d158e24ff2
1211 08:57:33.526144 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:33.527249 (+ 1105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1502637}
I20251211 08:57:33.528440 14905 tablet_replica.cc:333] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:57:33.528478 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 2cd6e4352cfc41e497dc58d158e24ff2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cd6e4352cfc41e497dc58d158e24ff2 already in progress: creating tablet
I20251211 08:57:33.532568 14905 raft_consensus.cc:2243] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:33.532976 14905 raft_consensus.cc:2272] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.534744 14905 ts_tablet_manager.cc:1916] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.333344 14714 raft_consensus.cc:359] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.535209 14714 raft_consensus.cc:385] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:33.535449 14714 raft_consensus.cc:740] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:33.535950 14714 consensus_queue.cc:260] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.563297 14577 tablet_service.cc:1505] Processing CreateTablet for tablet d4f826d079774152a3b66bfb81f990a8 (DEFAULT_TABLE table=default.Table_d1d406b278304b5ca6e6cc3b63e7fd36 [id=4208ab42111742f38028caeb7913a1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.564453 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f826d079774152a3b66bfb81f990a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.589699 14429 tablet_service.cc:1505] Processing CreateTablet for tablet 78c6a703e1e44f41931fe465def01bf6 (DEFAULT_TABLE table=default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.590710 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c6a703e1e44f41931fe465def01bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.596447 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f627382c623a4f818caf527706e776ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:33.597872 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f627382c623a4f818caf527706e776ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:33.599531 14272 tablet_service.cc:1505] Processing CreateTablet for tablet 78c6a703e1e44f41931fe465def01bf6 (DEFAULT_TABLE table=default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.599442 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_fe86659b8ddb426ab2ba622d58daab57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:33.600528 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c6a703e1e44f41931fe465def01bf6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:33.600780 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe86659b8ddb426ab2ba622d58daab57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:33.612408 15116 tablet.cc:1722] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:33.616278 15030 tablet_bootstrap.cc:492] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:33.616650 15030 ts_tablet_manager.cc:1403] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.134s user 0.007s sys 0.008s
I20251211 08:57:33.618935 15030 raft_consensus.cc:359] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.619503 15030 raft_consensus.cc:385] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:33.619737 15030 raft_consensus.cc:740] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:33.620309 15030 consensus_queue.cc:260] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.624713 14581 tablet_service.cc:1505] Processing CreateTablet for tablet 78c6a703e1e44f41931fe465def01bf6 (DEFAULT_TABLE table=default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.625648 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c6a703e1e44f41931fe465def01bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.625916 14714 ts_tablet_manager.cc:1434] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.306s user 0.006s sys 0.000s
I20251211 08:57:33.626691 14714 tablet_bootstrap.cc:492] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:33.627455 14273 tablet_service.cc:1505] Processing CreateTablet for tablet db302282bc3d4d8cb56d1cbb2f73c4fe (DEFAULT_TABLE table=default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.628392 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db302282bc3d4d8cb56d1cbb2f73c4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.631105 14722 tablet_bootstrap.cc:492] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:33.631489 14722 ts_tablet_manager.cc:1403] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.554s user 0.008s sys 0.003s
I20251211 08:57:33.633713 14722 raft_consensus.cc:359] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.634302 14722 raft_consensus.cc:385] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:33.641780 15030 ts_tablet_manager.cc:1434] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20251211 08:57:33.656071 15030 tablet_bootstrap.cc:492] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:33.660848 15119 tablet.cc:1722] T 559f9d428b5841708999725b5b859660 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:33.634541 14722 raft_consensus.cc:740] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:33.667220 14722 consensus_queue.cc:260] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.669525 14722 ts_tablet_manager.cc:1434] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.038s user 0.002s sys 0.003s
I20251211 08:57:33.670285 15083 raft_consensus.cc:493] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:33.670702 15083 raft_consensus.cc:515] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.672411 15083 leader_election.cc:290] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:33.673642 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15f7bcd680064b9f8fe51aa05acce6fb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:33.673749 14385 leader_election.cc:343] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Not found: Tablet not found: 15f7bcd680064b9f8fe51aa05acce6fb
I20251211 08:57:33.674196 14604 raft_consensus.cc:2468] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:33.675067 14382 leader_election.cc:304] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:33.676501 15083 raft_consensus.cc:2804] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:33.676836 15083 raft_consensus.cc:493] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:33.677188 15083 raft_consensus.cc:3060] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:33.677655 14427 tablet_service.cc:1505] Processing CreateTablet for tablet db302282bc3d4d8cb56d1cbb2f73c4fe (DEFAULT_TABLE table=default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.678639 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db302282bc3d4d8cb56d1cbb2f73c4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.685240 14584 tablet_service.cc:1505] Processing CreateTablet for tablet db302282bc3d4d8cb56d1cbb2f73c4fe (DEFAULT_TABLE table=default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.686205 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db302282bc3d4d8cb56d1cbb2f73c4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.687014 15030 tablet_bootstrap.cc:654] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:33.687467 15083 raft_consensus.cc:515] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:33.690653 14385 leader_election.cc:343] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Not found: Tablet not found: 15f7bcd680064b9f8fe51aa05acce6fb
I20251211 08:57:33.703202 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet a4e307e4d1994af0b99a88b3b06b73a4
I20251211 08:57:33.704583 14722 tablet_bootstrap.cc:492] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:33.706732 14013 catalog_manager.cc:6724] ReplaceTablet: tablet a4e307e4d1994af0b99a88b3b06b73a4 deleted and replaced by tablet 4eec7694087145d8a49dd93904f95980
I20251211 08:57:33.707008 14268 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e307e4d1994af0b99a88b3b06b73a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec7694087145d8a49dd93904f95980) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:33.707091 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e307e4d1994af0b99a88b3b06b73a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec7694087145d8a49dd93904f95980) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.709681 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e307e4d1994af0b99a88b3b06b73a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec7694087145d8a49dd93904f95980) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:33.720083 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15f7bcd680064b9f8fe51aa05acce6fb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:33.720587 14604 raft_consensus.cc:3060] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:33.722707 15030 tablet_bootstrap.cc:492] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:33.723157 15030 ts_tablet_manager.cc:1403] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.007s
I20251211 08:57:33.724050 14011 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet c181adf9db574ab39432b2f9b14fe4d7
I20251211 08:57:33.725286 14011 catalog_manager.cc:6724] ReplaceTablet: tablet c181adf9db574ab39432b2f9b14fe4d7 deleted and replaced by tablet f938359354b545cfbac560b7d7888b9f
I20251211 08:57:33.726850 14604 raft_consensus.cc:2468] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:33.727844 14382 leader_election.cc:304] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:33.728372 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet c181adf9db574ab39432b2f9b14fe4d7 with delete_type TABLET_DATA_DELETED (replaced by tablet f938359354b545cfbac560b7d7888b9f) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.729015 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet c181adf9db574ab39432b2f9b14fe4d7 with delete_type TABLET_DATA_DELETED (replaced by tablet f938359354b545cfbac560b7d7888b9f) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:33.726533 14007 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 1a2dcb9ac62e4aac885d638f4e2b072e (table default.Table_0c2ec37766b24f539c7ff1e0aa4ac9c0 [id=1673af33c2f846c7bf092303eff6e8ad]). Expected version 3 got 2
I20251211 08:57:33.729892 14007 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 6de43c843ff845a0bd4eb98b6054dd87 (table default.Table_3c7e457251e74eb7b8415bbcb6b6d7db [id=e5ffdeadfa904d85b68d0925da7e7a4c]). Expected version 1 got 0
I20251211 08:57:33.730604 14011 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ac11e9583ef64c1bbae0b2940213b819" } modify_external_catalogs: true
I20251211 08:57:33.731089 14011 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ac11e9583ef64c1bbae0b2940213b819" } modify_external_catalogs: true
I20251211 08:57:33.731164 14007 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 559f9d428b5841708999725b5b859660 (table default.Table_84c8a91dc6244ac0b8dd3a2986479da6 [id=3f3f000cd93e44df9f084873ba4b168b]). Expected version 1 got 0
I20251211 08:57:33.731904 14007 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 5cf1210733754f43885a7c0a350aa206 (table default.Table_a939f5e4faf546148710852f96ded5d8 [id=e9973d6c9b1341d1b96f04367e44e212]). Expected version 1 got 0
I20251211 08:57:33.732556 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet e961c5a3b5f640cbaea78b6185048d04
I20251211 08:57:33.734125 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7ca1c0fc20ca4dd7a11793c0ac968666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:33.738104 14718 ts_tablet_manager.cc:1929] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:33.738413 14718 log.cc:1199] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/426e26859b9c4966b8ec8ba3eab53a43
I20251211 08:57:33.739025 15083 leader_election.cc:290] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:33.739269 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet e961c5a3b5f640cbaea78b6185048d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.739459 15083 raft_consensus.cc:2804] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:33.739818 15083 raft_consensus.cc:697] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:33.739910 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet e961c5a3b5f640cbaea78b6185048d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:33.740528 15083 consensus_queue.cc:237] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:33.740980 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ca1c0fc20ca4dd7a11793c0ac968666 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:33.744113 14714 tablet_bootstrap.cc:654] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:33.744015 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 7b90b0c944174b2a82696e3fee806934 (DEFAULT_TABLE table=default.Table_e3e740f7ffee4b1683b5888496b1a384 [id=0f8c030be5504d859ef25176b6cbb4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.744998 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b90b0c944174b2a82696e3fee806934. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.745255 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 7b90b0c944174b2a82696e3fee806934 (DEFAULT_TABLE table=default.Table_e3e740f7ffee4b1683b5888496b1a384 [id=0f8c030be5504d859ef25176b6cbb4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.746150 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b90b0c944174b2a82696e3fee806934. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.748332 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 174fefd70bcc41418a77a5716f97cd5f (DEFAULT_TABLE table=default.Table_fcf104ee591d470d806156189c2cac52 [id=b734e827456d451b9fe6f1dbf47fea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.749249 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 174fefd70bcc41418a77a5716f97cd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.750865 14718 ts_tablet_manager.cc:1950] T 426e26859b9c4966b8ec8ba3eab53a43 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:33.752007 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 174fefd70bcc41418a77a5716f97cd5f (DEFAULT_TABLE table=default.Table_fcf104ee591d470d806156189c2cac52 [id=b734e827456d451b9fe6f1dbf47fea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.752964 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 174fefd70bcc41418a77a5716f97cd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.753064 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 426e26859b9c4966b8ec8ba3eab53a43 (table default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6]) successfully deleted
I20251211 08:57:33.760591 14269 tablet_service.cc:1505] Processing CreateTablet for tablet 7b90b0c944174b2a82696e3fee806934 (DEFAULT_TABLE table=default.Table_e3e740f7ffee4b1683b5888496b1a384 [id=0f8c030be5504d859ef25176b6cbb4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.760811 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 174fefd70bcc41418a77a5716f97cd5f (DEFAULT_TABLE table=default.Table_fcf104ee591d470d806156189c2cac52 [id=b734e827456d451b9fe6f1dbf47fea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.762308 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet c181adf9db574ab39432b2f9b14fe4d7 with delete_type TABLET_DATA_DELETED (replaced by tablet f938359354b545cfbac560b7d7888b9f) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:33.764859 14714 tablet_bootstrap.cc:492] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:33.765226 14714 ts_tablet_manager.cc:1403] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.139s user 0.008s sys 0.000s
I20251211 08:57:33.766954 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 174fefd70bcc41418a77a5716f97cd5f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:33.771293 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41d6ef975a47489385791065bb8f9f46 consensus state has no leader
I20251211 08:57:33.772368 14931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.13.146.62:39015,127.13.146.61:34977,127.13.146.60:38671
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.62:39015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.146.62
--webserver_port=43581
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:57:33.774021 14931 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20251211 08:57:33.775846 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5cf1210733754f43885a7c0a350aa206 consensus state has no leader
I20251211 08:57:33.777416 14718 tablet_replica.cc:333] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:33.778236 14718 raft_consensus.cc:2243] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:33.780325 14904 ts_tablet_manager.cc:1929] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:33.780627 14904 log.cc:1199] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/292b641df22f478d8849ff6148ecf951
I20251211 08:57:33.781247 14904 ts_tablet_manager.cc:1950] T 292b641df22f478d8849ff6148ecf951 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:33.767061 14268 tablet_service.cc:1552] Processing DeleteTablet for tablet e961c5a3b5f640cbaea78b6185048d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:33.789337 14718 raft_consensus.cc:2272] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.791699 14714 raft_consensus.cc:359] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.792294 14714 raft_consensus.cc:385] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:33.792507 14714 raft_consensus.cc:740] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:33.792937 14714 consensus_queue.cc:260] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.793931 14718 ts_tablet_manager.cc:1916] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.802018 15030 raft_consensus.cc:359] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.803061 15030 raft_consensus.cc:385] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:33.803375 15030 raft_consensus.cc:740] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:33.806090 14714 ts_tablet_manager.cc:1434] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20251211 08:57:33.812026 14904 tablet_replica.cc:333] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:33.812958 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 25805a95349d49f58280a30e6876929d with delete_type TABLET_DATA_DELETED (replaced by tablet 41d6ef975a47489385791065bb8f9f46) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:33.818543 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 292b641df22f478d8849ff6148ecf951 (table default.Table_adee98d887af4b9bb8d59ea922bd36f9 [id=1d8b25ddadc64095ad0422142f43812b]) successfully deleted
I20251211 08:57:33.819221 14904 raft_consensus.cc:2243] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:33.806905 14714 tablet_bootstrap.cc:492] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:33.824961 14904 pending_rounds.cc:70] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
I20251211 08:57:33.825168 14904 pending_rounds.cc:77] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256383912976384 op_type: NO_OP noop_request { }
I20251211 08:57:33.825449 14904 raft_consensus.cc:2889] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:33.825711 14904 raft_consensus.cc:2272] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.829340 14722 tablet_bootstrap.cc:654] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:33.831801 14718 ts_tablet_manager.cc:1929] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:33.832158 14718 log.cc:1199] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/292b641df22f478d8849ff6148ecf951
I20251211 08:57:33.832913 14718 ts_tablet_manager.cc:1950] T 292b641df22f478d8849ff6148ecf951 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:33.804088 15030 consensus_queue.cc:260] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.834939 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 8ad600554c0946e0bcb0da6a42be2ffd (DEFAULT_TABLE table=default.Table_ccb3af786cad4564b032d78520da1cc3 [id=62e4ec0c216b41eebee5490eaba6ef5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.835532 14714 tablet_bootstrap.cc:654] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:33.835894 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad600554c0946e0bcb0da6a42be2ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.843890 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 8ad600554c0946e0bcb0da6a42be2ffd (DEFAULT_TABLE table=default.Table_ccb3af786cad4564b032d78520da1cc3 [id=62e4ec0c216b41eebee5490eaba6ef5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.844244 14279 tablet_service.cc:1505] Processing CreateTablet for tablet a6570281e1064c79b6e2297723f28658 (DEFAULT_TABLE table=default.Table_ba58b6955e7b4ab6ba45e12826d6f56d [id=45f67c67257e428f8ed2f16898ad5fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.851871 14268 tablet_service.cc:1552] Processing DeleteTablet for tablet 60b23163906447c3877db631968f5623 with delete_type TABLET_DATA_DELETED (replaced by tablet e961c5a3b5f640cbaea78b6185048d04) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:33.828361 14272 tablet_service.cc:1552] Processing DeleteTablet for tablet 426e26859b9c4966b8ec8ba3eab53a43 with delete_type TABLET_DATA_DELETED (replaced by tablet e15483f539f24f1ba1b442663d66fd45) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:33.859054 15030 ts_tablet_manager.cc:1434] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.136s user 0.006s sys 0.000s
I20251211 08:57:33.859977 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 113) took 1045 ms. Trace:
I20251211 08:57:33.858394 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b90b0c944174b2a82696e3fee806934. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.860276 14718 rpcz_store.cc:276] 1211 08:57:32.814817 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.814936 (+ 119us) service_pool.cc:225] Handling call
1211 08:57:33.777270 (+962334us) ts_tablet_manager.cc:1151] Deleting tablet 292b641df22f478d8849ff6148ecf951
1211 08:57:33.777307 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:33.817987 (+ 40680us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:33.831610 (+ 13623us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:33.859912 (+ 28302us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10636,"tablet-delete.queue_time_us":949595}
I20251211 08:57:33.861409 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 292b641df22f478d8849ff6148ecf951 (table default.Table_adee98d887af4b9bb8d59ea922bd36f9 [id=1d8b25ddadc64095ad0422142f43812b]) successfully deleted
I20251211 08:57:33.861578 14718 tablet_replica.cc:333] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:33.862236 14718 raft_consensus.cc:2243] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:33.862636 14718 raft_consensus.cc:2272] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.864874 14714 tablet_bootstrap.cc:492] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:33.865242 14714 ts_tablet_manager.cc:1403] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.000s
I20251211 08:57:33.867385 14714 raft_consensus.cc:359] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.867980 14714 raft_consensus.cc:385] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:33.868216 14714 raft_consensus.cc:740] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:33.868029 14931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251211 08:57:33.868721 14714 consensus_queue.cc:260] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.869666 14718 ts_tablet_manager.cc:1916] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.872124 14010 catalog_manager.cc:5654] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:33.875702 15030 tablet_bootstrap.cc:492] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:33.877962 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 8ad600554c0946e0bcb0da6a42be2ffd (DEFAULT_TABLE table=default.Table_ccb3af786cad4564b032d78520da1cc3 [id=62e4ec0c216b41eebee5490eaba6ef5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.879940 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad600554c0946e0bcb0da6a42be2ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.883286 14714 ts_tablet_manager.cc:1434] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20251211 08:57:33.885864 14714 tablet_bootstrap.cc:492] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:33.891207 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6570281e1064c79b6e2297723f28658. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.897439 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad600554c0946e0bcb0da6a42be2ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.910763 14714 tablet_bootstrap.cc:654] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:33.857640 14905 ts_tablet_manager.cc:1929] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:33.911728 14905 log.cc:1199] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/25805a95349d49f58280a30e6876929d
I20251211 08:57:33.912549 14905 ts_tablet_manager.cc:1950] T 25805a95349d49f58280a30e6876929d P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:33.913985 14904 ts_tablet_manager.cc:1916] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.914366 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 89) took 1665 ms. Trace:
I20251211 08:57:33.914659 14905 rpcz_store.cc:276] 1211 08:57:32.249064 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.249491 (+ 427us) service_pool.cc:225] Handling call
1211 08:57:33.528318 (+1278827us) ts_tablet_manager.cc:1151] Deleting tablet 25805a95349d49f58280a30e6876929d
1211 08:57:33.528349 (+ 31us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:33.752801 (+224452us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:33.857303 (+104502us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:33.914302 (+ 56999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1247761}
I20251211 08:57:33.916673 14424 tablet_service.cc:1505] Processing CreateTablet for tablet a6570281e1064c79b6e2297723f28658 (DEFAULT_TABLE table=default.Table_ba58b6955e7b4ab6ba45e12826d6f56d [id=45f67c67257e428f8ed2f16898ad5fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.917572 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6570281e1064c79b6e2297723f28658. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.926939 14011 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet caf537c461c04a81b66f268ee98dc64e
I20251211 08:57:33.929078 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet caf537c461c04a81b66f268ee98dc64e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:33.942703 14005 master_service.cc:954] ReplaceTablet: received request to replace tablet 3c5315f708c04322919c714dc915a483 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:33.945530 14011 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d429fb0e546d4f3d886de3a04af241cf" } modify_external_catalogs: true
I20251211 08:57:33.945952 14011 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d429fb0e546d4f3d886de3a04af241cf" } modify_external_catalogs: true
I20251211 08:57:33.948644 14718 ts_tablet_manager.cc:1929] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:33.950914 14718 log.cc:1199] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/7b9b338a3f544ccb813247250b0f0eb7
I20251211 08:57:33.936671 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet caf537c461c04a81b66f268ee98dc64e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:33.954267 14429 tablet_service.cc:1505] Processing CreateTablet for tablet e961c5a3b5f640cbaea78b6185048d04 (DEFAULT_TABLE table=default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.960079 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e961c5a3b5f640cbaea78b6185048d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.966101 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_96d05b744df34836be68b4841e6c2a2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:33.967321 14718 ts_tablet_manager.cc:1950] T 7b9b338a3f544ccb813247250b0f0eb7 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
W20251211 08:57:33.967446 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96d05b744df34836be68b4841e6c2a2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:33.968231 14714 tablet_bootstrap.cc:492] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:33.968575 14714 ts_tablet_manager.cc:1403] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.083s user 0.009s sys 0.002s
I20251211 08:57:33.969533 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 7b9b338a3f544ccb813247250b0f0eb7 (table default.Table_f0b7cb6abf9b49fd8b0051bb2eb9e55e [id=85b5244918f44d04a067fe1aa8b35975]) successfully deleted
I20251211 08:57:33.970750 14714 raft_consensus.cc:359] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:33.970993 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 7b60d99c2b4f442da4a4d9c5190f5797 because the tablet was not found. No further retry: Not found: Tablet not found: 7b60d99c2b4f442da4a4d9c5190f5797
I20251211 08:57:33.971428 14714 raft_consensus.cc:385] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:33.971663 14714 raft_consensus.cc:740] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:33.972220 14714 consensus_queue.cc:260] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:33.973378 14718 tablet_replica.cc:333] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:33.991694 14718 raft_consensus.cc:2243] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:33.992158 14718 raft_consensus.cc:2272] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:33.993243 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ba58b6955e7b4ab6ba45e12826d6f56d" } new_table_name: "default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b" modify_external_catalogs: true
I20251211 08:57:33.993700 14718 ts_tablet_manager.cc:1916] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:33.995472 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_2670ab0571da4c8f8ccbeee370ba3914" } new_table_name: "default.Table_f0670566e5df45448e1786b9c82156d4" modify_external_catalogs: true
I20251211 08:57:33.992230 14429 tablet_service.cc:1505] Processing CreateTablet for tablet 41d6ef975a47489385791065bb8f9f46 (DEFAULT_TABLE table=default.Table_fa2830322a8a4f7b96cd19543b967b05 [id=52ae4637827d424eb4d37362bf21cea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:33.996732 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41d6ef975a47489385791065bb8f9f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.999286 14714 ts_tablet_manager.cc:1434] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20251211 08:57:34.000154 14714 tablet_bootstrap.cc:492] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:34.008360 14272 tablet_service.cc:1505] Processing CreateTablet for tablet 41d6ef975a47489385791065bb8f9f46 (DEFAULT_TABLE table=default.Table_fa2830322a8a4f7b96cd19543b967b05 [id=52ae4637827d424eb4d37362bf21cea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:34.009349 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41d6ef975a47489385791065bb8f9f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:33.930245 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet caf537c461c04a81b66f268ee98dc64e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:33.938935 15089 raft_consensus.cc:493] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:34.015111 15089 raft_consensus.cc:515] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.016708 15089 leader_election.cc:290] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:33.950170 14581 tablet_service.cc:1505] Processing CreateTablet for tablet a6570281e1064c79b6e2297723f28658 (DEFAULT_TABLE table=default.Table_ba58b6955e7b4ab6ba45e12826d6f56d [id=45f67c67257e428f8ed2f16898ad5fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:34.017982 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e410da32b3446edbf84b2eaa96424c3" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:33.985467 14584 tablet_service.cc:1505] Processing CreateTablet for tablet e961c5a3b5f640cbaea78b6185048d04 (DEFAULT_TABLE table=default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:34.018463 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6570281e1064c79b6e2297723f28658. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:34.019291 14542 leader_election.cc:343] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:34.001384 14577 tablet_service.cc:1505] Processing CreateTablet for tablet 41d6ef975a47489385791065bb8f9f46 (DEFAULT_TABLE table=default.Table_fa2830322a8a4f7b96cd19543b967b05 [id=52ae4637827d424eb4d37362bf21cea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:34.020541 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e961c5a3b5f640cbaea78b6185048d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:34.023252 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41d6ef975a47489385791065bb8f9f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:34.023694 14714 tablet_bootstrap.cc:654] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.028455 14718 ts_tablet_manager.cc:1929] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:34.028860 14718 log.cc:1199] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/d235bade012540e1bb1c0dd871eff1bb
I20251211 08:57:34.031517 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 25805a95349d49f58280a30e6876929d (table default.Table_fa2830322a8a4f7b96cd19543b967b05 [id=52ae4637827d424eb4d37362bf21cea2]) successfully deleted
I20251211 08:57:34.032371 14905 tablet_replica.cc:333] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:34.033113 14905 raft_consensus.cc:2243] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:34.033259 14718 ts_tablet_manager.cc:1950] T d235bade012540e1bb1c0dd871eff1bb P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:34.033596 14905 raft_consensus.cc:2272] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:34.035249 14905 ts_tablet_manager.cc:1916] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:34.044158 14714 tablet_bootstrap.cc:492] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:34.044559 14714 ts_tablet_manager.cc:1403] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.002s
I20251211 08:57:34.046587 14714 raft_consensus.cc:359] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.047240 14714 raft_consensus.cc:385] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.047483 14714 raft_consensus.cc:740] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.048435 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet d235bade012540e1bb1c0dd871eff1bb (table default.Table_7c70be2e41de4127a511a479fdb6ad7e [id=5b6d7c4fd998498f91fe1594382caed9]) successfully deleted
I20251211 08:57:34.049356 14718 tablet_replica.cc:333] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:34.050015 14718 raft_consensus.cc:2243] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:34.053830 14718 pending_rounds.cc:70] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:34.054052 14718 pending_rounds.cc:77] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256378164461568 op_type: NO_OP noop_request { }
I20251211 08:57:34.054378 14718 raft_consensus.cc:2889] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:34.055887 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_d40cb8a6b4974e2e9ff982e893a2c552"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:34.054643 14718 raft_consensus.cc:2272] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:34.057300 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d40cb8a6b4974e2e9ff982e893a2c552 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:34.058498 14718 ts_tablet_manager.cc:1916] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:34.059340 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_8363e9c4cd624dbe8ba7edf7ac61591d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:34.060573 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8363e9c4cd624dbe8ba7edf7ac61591d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:34.067133 14013 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_0c2ec37766b24f539c7ff1e0aa4ac9c0 [id=1673af33c2f846c7bf092303eff6e8ad] alter complete (version 3)
I20251211 08:57:34.055344 15030 tablet_bootstrap.cc:654] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.063144 14714 consensus_queue.cc:260] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.077039 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e410da32b3446edbf84b2eaa96424c3" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:34.078475 14714 ts_tablet_manager.cc:1434] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
W20251211 08:57:34.078557 14544 leader_election.cc:343] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:34.078927 14544 leader_election.cc:304] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:34.079303 14714 tablet_bootstrap.cc:492] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:34.087787 15089 raft_consensus.cc:2749] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:34.090569 14448 raft_consensus.cc:3060] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:34.092732 15089 raft_consensus.cc:493] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:34.093173 15089 raft_consensus.cc:515] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.094882 15089 leader_election.cc:290] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:34.096170 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6e533e1f4da7b03a800d029f712d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:34.097641 14542 leader_election.cc:343] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:34.099246 15038 consensus_queue.cc:1048] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:34.100572 13997 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 41d6ef975a47489385791065bb8f9f46 (table default.Table_fa2830322a8a4f7b96cd19543b967b05 [id=52ae4637827d424eb4d37362bf21cea2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 41d6ef975a47489385791065bb8f9f46 consensus state has no leader
I20251211 08:57:34.100811 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6e533e1f4da7b03a800d029f712d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:34.101547 14271 tablet_service.cc:1505] Processing CreateTablet for tablet e961c5a3b5f640cbaea78b6185048d04 (DEFAULT_TABLE table=default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:34.107679 14714 tablet_bootstrap.cc:654] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.113992 14718 ts_tablet_manager.cc:1929] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:34.124262 14718 log.cc:1199] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/38655ee4996e41cc8f87b5248264613d
I20251211 08:57:34.125411 14718 ts_tablet_manager.cc:1950] T 38655ee4996e41cc8f87b5248264613d P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
W20251211 08:57:34.126812 14544 leader_election.cc:343] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:34.127296 14544 leader_election.cc:304] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:34.128261 15089 raft_consensus.cc:2749] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:34.127974 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 118) took 1039 ms. Trace:
I20251211 08:57:34.130483 14718 rpcz_store.cc:276] 1211 08:57:33.088308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.088451 (+ 143us) service_pool.cc:225] Handling call
1211 08:57:34.049222 (+960771us) ts_tablet_manager.cc:1151] Deleting tablet 38655ee4996e41cc8f87b5248264613d
1211 08:57:34.049261 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.098686 (+ 49425us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.113786 (+ 15100us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.127925 (+ 14139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":960233}
I20251211 08:57:34.135918 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 38655ee4996e41cc8f87b5248264613d (table default.Table_a7ade9d81bfc4498a43e0b5ad6941ac2 [id=c533026c25a54aa885eb483d4120f720]) successfully deleted
I20251211 08:57:34.136960 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 119) took 1046 ms. Trace:
I20251211 08:57:34.137419 15030 tablet_bootstrap.cc:492] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:34.137792 15030 ts_tablet_manager.cc:1403] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.262s user 0.010s sys 0.000s
I20251211 08:57:34.137924 14718 rpcz_store.cc:276] 1211 08:57:33.090281 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.090403 (+ 122us) service_pool.cc:225] Handling call
1211 08:57:34.136748 (+1046345us) ts_tablet_manager.cc:1151] Deleting tablet 1853070f81c34b52b8c74a7402e6dad3
1211 08:57:34.136783 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.136916 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045823}
W20251211 08:57:34.139587 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 1853070f81c34b52b8c74a7402e6dad3 because the tablet was not found. No further retry: Not found: Tablet not found: 1853070f81c34b52b8c74a7402e6dad3
W20251211 08:57:34.141665 14542 consensus_peers.cc:597] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 292b641df22f478d8849ff6148ecf951. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:34.141496 15030 raft_consensus.cc:359] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.142119 15030 raft_consensus.cc:385] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.142364 15030 raft_consensus.cc:740] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.143149 14718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 120) took 1013 ms. Trace:
I20251211 08:57:34.143530 14718 rpcz_store.cc:276] 1211 08:57:33.130142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.130256 (+ 114us) service_pool.cc:225] Handling call
1211 08:57:34.142906 (+1012650us) ts_tablet_manager.cc:1151] Deleting tablet 00f79094ed8b4782a3a1488e41794f9b
1211 08:57:34.142959 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.143100 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1011867}
W20251211 08:57:34.144634 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 00f79094ed8b4782a3a1488e41794f9b because the tablet was not found. No further retry: Not found: Tablet not found: 00f79094ed8b4782a3a1488e41794f9b
I20251211 08:57:34.145628 14718 tablet_replica.cc:333] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
W20251211 08:57:34.147027 14382 consensus_peers.cc:597] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e015d0fb9d6c4c80a510c05244b334e4. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:34.148424 14714 tablet_bootstrap.cc:492] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:34.148869 14714 ts_tablet_manager.cc:1403] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.070s user 0.015s sys 0.000s
I20251211 08:57:34.150982 14714 raft_consensus.cc:359] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.151548 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e961c5a3b5f640cbaea78b6185048d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:34.151633 14714 raft_consensus.cc:385] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.151973 14714 raft_consensus.cc:740] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.152588 14714 consensus_queue.cc:260] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.153472 14718 raft_consensus.cc:2243] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:34.158807 14718 pending_rounds.cc:70] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:34.159060 14718 pending_rounds.cc:77] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256384141856768 op_type: NO_OP noop_request { }
I20251211 08:57:34.159775 14718 raft_consensus.cc:2889] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:34.160102 14718 raft_consensus.cc:2272] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:34.160334 14714 ts_tablet_manager.cc:1434] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20251211 08:57:34.161404 14714 tablet_bootstrap.cc:492] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:34.164394 14718 ts_tablet_manager.cc:1916] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:34.172178 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 3c5315f708c04322919c714dc915a483
I20251211 08:57:34.173346 14005 catalog_manager.cc:6724] ReplaceTablet: tablet 3c5315f708c04322919c714dc915a483 deleted and replaced by tablet 69aae3e1d7f34c70b0b48ce63089b482
I20251211 08:57:34.173523 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c5315f708c04322919c714dc915a483 with delete_type TABLET_DATA_DELETED (replaced by tablet 69aae3e1d7f34c70b0b48ce63089b482) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:34.174072 14005 master_service.cc:954] ReplaceTablet: received request to replace tablet 5daba165262e41af8d3481d3b4929092 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:34.177080 14714 tablet_bootstrap.cc:654] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.185798 15030 consensus_queue.cc:260] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.190840 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c5315f708c04322919c714dc915a483 with delete_type TABLET_DATA_DELETED (replaced by tablet 69aae3e1d7f34c70b0b48ce63089b482) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:34.191648 15122 raft_consensus.cc:493] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:34.192050 15122 raft_consensus.cc:515] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.193629 15122 leader_election.cc:290] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:34.195029 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cf1210733754f43885a7c0a350aa206" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:34.195564 14606 raft_consensus.cc:2468] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:34.197054 14382 leader_election.cc:304] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:34.197952 15122 raft_consensus.cc:2804] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:34.198236 15122 raft_consensus.cc:493] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:34.198493 15122 raft_consensus.cc:3060] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:34.202598 14718 ts_tablet_manager.cc:1929] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:34.202932 14718 log.cc:1199] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/e015d0fb9d6c4c80a510c05244b334e4
I20251211 08:57:34.203537 14718 ts_tablet_manager.cc:1950] T e015d0fb9d6c4c80a510c05244b334e4 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
W20251211 08:57:34.203701 15136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:57:34.205619 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet e015d0fb9d6c4c80a510c05244b334e4 (table default.Table_f647a2c75e7c46939a62573dea122fef [id=779be79e440a45d7ac4f6afb05bed936]) successfully deleted
I20251211 08:57:34.206182 14718 tablet_replica.cc:333] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:34.206764 14718 raft_consensus.cc:2243] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
W20251211 08:57:34.206768 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet c5191443773b415198687c55c4517692 because the tablet was not found. No further retry: Not found: Tablet not found: c5191443773b415198687c55c4517692
W20251211 08:57:34.207340 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet a4e307e4d1994af0b99a88b3b06b73a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4e307e4d1994af0b99a88b3b06b73a4 already in progress: creating tablet
I20251211 08:57:34.207481 14718 raft_consensus.cc:2272] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:34.208215 15030 ts_tablet_manager.cc:1434] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.070s user 0.005s sys 0.001s
I20251211 08:57:34.208985 14718 ts_tablet_manager.cc:1916] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:34.212306 15122 raft_consensus.cc:515] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.225303 15030 tablet_bootstrap.cc:492] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:34.230325 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cf1210733754f43885a7c0a350aa206" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:34.230862 14606 raft_consensus.cc:3060] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:34.234174 14718 ts_tablet_manager.cc:1929] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:34.234473 14718 log.cc:1199] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/c181adf9db574ab39432b2f9b14fe4d7
I20251211 08:57:34.237349 14606 raft_consensus.cc:2468] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:34.238446 14382 leader_election.cc:304] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:34.240355 15083 raft_consensus.cc:2804] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:34.240804 15083 raft_consensus.cc:697] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:34.241595 15083 consensus_queue.cc:237] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.235100 14718 ts_tablet_manager.cc:1950] T c181adf9db574ab39432b2f9b14fe4d7 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:34.249441 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet c181adf9db574ab39432b2f9b14fe4d7 (table default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]) successfully deleted
W20251211 08:57:34.251420 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet e961c5a3b5f640cbaea78b6185048d04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e961c5a3b5f640cbaea78b6185048d04 already in progress: creating tablet
I20251211 08:57:34.251554 15122 leader_election.cc:290] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:34.253495 14011 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet a0f9c321e673433498a9059dd2e2efe6
I20251211 08:57:34.259830 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet a0f9c321e673433498a9059dd2e2efe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:34.262668 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d12516554d1547419ff004d016cb9a3a" } modify_external_catalogs: true
I20251211 08:57:34.263206 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d12516554d1547419ff004d016cb9a3a" } modify_external_catalogs: true
I20251211 08:57:34.267671 15030 tablet_bootstrap.cc:654] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:34.268008 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet caf537c461c04a81b66f268ee98dc64e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caf537c461c04a81b66f268ee98dc64e already in progress: creating tablet
W20251211 08:57:34.268689 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 3c5315f708c04322919c714dc915a483 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c5315f708c04322919c714dc915a483 already in progress: creating tablet
I20251211 08:57:34.269667 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet a0f9c321e673433498a9059dd2e2efe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:34.270370 14718 tablet_replica.cc:333] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:34.271296 14718 raft_consensus.cc:2243] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:34.272248 14718 raft_consensus.cc:2272] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:34.273427 14714 tablet_bootstrap.cc:492] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:34.273778 14714 ts_tablet_manager.cc:1403] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.113s user 0.008s sys 0.001s
I20251211 08:57:34.273854 14718 ts_tablet_manager.cc:1916] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:34.273957 14905 ts_tablet_manager.cc:1929] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:34.274353 14905 log.cc:1199] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/15f7bcd680064b9f8fe51aa05acce6fb
I20251211 08:57:34.275005 14905 ts_tablet_manager.cc:1950] T 15f7bcd680064b9f8fe51aa05acce6fb P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:34.294148 14718 ts_tablet_manager.cc:1929] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:34.294545 14718 log.cc:1199] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/a0f9c321e673433498a9059dd2e2efe6
I20251211 08:57:34.295315 14718 ts_tablet_manager.cc:1950] T a0f9c321e673433498a9059dd2e2efe6 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:34.276744 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 90) took 2027 ms. Trace:
I20251211 08:57:34.304605 14905 rpcz_store.cc:276] 1211 08:57:32.249364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.250028 (+ 664us) service_pool.cc:225] Handling call
1211 08:57:34.032237 (+1782209us) ts_tablet_manager.cc:1151] Deleting tablet 15f7bcd680064b9f8fe51aa05acce6fb
1211 08:57:34.032274 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.181534 (+149260us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.261475 (+ 79941us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.276695 (+ 15220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1751526}
I20251211 08:57:34.307327 15083 raft_consensus.cc:493] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:34.307780 15083 raft_consensus.cc:515] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.309333 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet a0f9c321e673433498a9059dd2e2efe6 (table default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5]) successfully deleted
I20251211 08:57:34.309639 15083 leader_election.cc:290] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:34.310372 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 15f7bcd680064b9f8fe51aa05acce6fb (table default.Table_d1d406b278304b5ca6e6cc3b63e7fd36 [id=4208ab42111742f38028caeb7913a1c7]) successfully deleted
I20251211 08:57:34.309367 15030 tablet_bootstrap.cc:492] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:34.311283 15030 ts_tablet_manager.cc:1403] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.086s user 0.009s sys 0.000s
I20251211 08:57:34.316907 15030 raft_consensus.cc:359] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:34.317149 14382 leader_election.cc:343] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 60b23163906447c3877db631968f5623
I20251211 08:57:34.276147 14714 raft_consensus.cc:359] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.321684 14714 raft_consensus.cc:385] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.322799 14714 raft_consensus.cc:740] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.324502 14714 consensus_queue.cc:260] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:34.331418 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6570281e1064c79b6e2297723f28658 consensus state has no leader
I20251211 08:57:34.332360 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a9bd9e30e1ee4c92bbdd8e1748dd0ddd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:34.331709 14714 ts_tablet_manager.cc:1434] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20251211 08:57:34.333734 14714 tablet_bootstrap.cc:492] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:34.334041 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9bd9e30e1ee4c92bbdd8e1748dd0ddd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:34.340972 15030 raft_consensus.cc:385] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.341861 15030 raft_consensus.cc:740] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.311885 14905 tablet_replica.cc:333] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:34.347075 14905 raft_consensus.cc:2243] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:34.347591 14905 raft_consensus.cc:2272] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:34.349285 14905 ts_tablet_manager.cc:1916] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:34.352113 15030 consensus_queue.cc:260] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:34.359891 14544 consensus_peers.cc:597] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 292b641df22f478d8849ff6148ecf951. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:34.365212 15030 ts_tablet_manager.cc:1434] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.050s user 0.003s sys 0.001s
I20251211 08:57:34.386442 15030 tablet_bootstrap.cc:492] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:34.403151 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 5daba165262e41af8d3481d3b4929092
I20251211 08:57:34.404359 14005 catalog_manager.cc:6724] ReplaceTablet: tablet 5daba165262e41af8d3481d3b4929092 deleted and replaced by tablet da52cace2aab4f9dacef18ec5fa56aaf
I20251211 08:57:34.406144 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet 5daba165262e41af8d3481d3b4929092 with delete_type TABLET_DATA_DELETED (replaced by tablet da52cace2aab4f9dacef18ec5fa56aaf) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:34.406854 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet 5daba165262e41af8d3481d3b4929092 with delete_type TABLET_DATA_DELETED (replaced by tablet da52cace2aab4f9dacef18ec5fa56aaf) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:34.409545 14722 tablet_bootstrap.cc:492] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:34.410271 14722 ts_tablet_manager.cc:1403] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.707s user 0.001s sys 0.018s
I20251211 08:57:34.412771 15030 tablet_bootstrap.cc:654] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:34.421095 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 5daba165262e41af8d3481d3b4929092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5daba165262e41af8d3481d3b4929092 already in progress: creating tablet
W20251211 08:57:34.419961 14230 consensus_peers.cc:597] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25805a95349d49f58280a30e6876929d. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:34.427232 14714 tablet_bootstrap.cc:654] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.428421 14904 ts_tablet_manager.cc:1929] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:34.428834 14904 log.cc:1199] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/d235bade012540e1bb1c0dd871eff1bb
I20251211 08:57:34.429724 14904 ts_tablet_manager.cc:1950] T d235bade012540e1bb1c0dd871eff1bb P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:34.456897 14714 tablet_bootstrap.cc:492] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:34.453387 14722 raft_consensus.cc:359] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.458490 14722 raft_consensus.cc:385] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.458932 14722 raft_consensus.cc:740] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.459692 14722 consensus_queue.cc:260] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.461524 14722 ts_tablet_manager.cc:1434] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.051s user 0.001s sys 0.007s
I20251211 08:57:34.464147 14905 ts_tablet_manager.cc:1929] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:34.464494 14905 log.cc:1199] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/a9524aa1503549bd820918d1c025aa2e
I20251211 08:57:34.465112 14905 ts_tablet_manager.cc:1950] T a9524aa1503549bd820918d1c025aa2e P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:34.465206 14006 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 69aae3e1d7f34c70b0b48ce63089b482
I20251211 08:57:34.466252 14714 ts_tablet_manager.cc:1403] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.133s user 0.011s sys 0.000s
I20251211 08:57:34.468322 14714 raft_consensus.cc:359] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.468923 14714 raft_consensus.cc:385] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.469154 14714 raft_consensus.cc:740] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.469493 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_c15367b621c144348ec8511d2947fec6" } modify_external_catalogs: true
I20251211 08:57:34.470000 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_c15367b621c144348ec8511d2947fec6" } modify_external_catalogs: true
I20251211 08:57:34.472559 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_44453967d1924a139aaf13c5030ace59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:34.473875 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44453967d1924a139aaf13c5030ace59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:34.484586 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cf1210733754f43885a7c0a350aa206" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:34.478904 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 111) took 1415 ms. Trace:
I20251211 08:57:34.486052 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cf1210733754f43885a7c0a350aa206" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:34.487389 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60b23163906447c3877db631968f5623" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:34.487990 14294 raft_consensus.cc:2468] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:34.485996 14904 rpcz_store.cc:276] 1211 08:57:33.063054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.063254 (+ 200us) service_pool.cc:225] Handling call
1211 08:57:33.811889 (+748635us) ts_tablet_manager.cc:1151] Deleting tablet d235bade012540e1bb1c0dd871eff1bb
1211 08:57:33.811930 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.164376 (+352446us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.428080 (+263704us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.478796 (+ 50716us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":748177}
I20251211 08:57:34.495311 14272 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c5315f708c04322919c714dc915a483 with delete_type TABLET_DATA_DELETED (replaced by tablet 69aae3e1d7f34c70b0b48ce63089b482) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:34.495539 15148 raft_consensus.cc:493] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:34.466710 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 104) took 1868 ms. Trace:
W20251211 08:57:34.498314 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 153978us
I20251211 08:57:34.498044 15148 raft_consensus.cc:515] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.501286 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccce5a446ac9460c9c9b7de72d875d95" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:34.501554 14904 tablet_replica.cc:333] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:34.501873 14606 raft_consensus.cc:2468] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:34.502282 14904 raft_consensus.cc:2243] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:34.502707 14904 raft_consensus.cc:2272] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:34.498351 14905 rpcz_store.cc:276] 1211 08:57:32.598458 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.598686 (+ 228us) service_pool.cc:225] Handling call
1211 08:57:34.311753 (+1713067us) ts_tablet_manager.cc:1151] Deleting tablet a9524aa1503549bd820918d1c025aa2e
1211 08:57:34.311789 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.400432 (+ 88643us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.463949 (+ 63517us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.466665 (+ 2716us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1663549}
I20251211 08:57:34.504645 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet a9524aa1503549bd820918d1c025aa2e (table default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]) successfully deleted
I20251211 08:57:34.505810 14905 tablet_replica.cc:333] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:34.509753 14905 raft_consensus.cc:2243] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:34.511924 14905 raft_consensus.cc:2272] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:34.517045 14905 ts_tablet_manager.cc:1916] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:34.504789 14230 leader_election.cc:304] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:34.497224 14722 tablet_bootstrap.cc:492] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:34.527842 15030 tablet_bootstrap.cc:492] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:34.528236 15030 ts_tablet_manager.cc:1403] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.157s user 0.008s sys 0.007s
I20251211 08:57:34.508190 15148 leader_election.cc:290] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:34.529070 15148 raft_consensus.cc:2804] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:34.529371 15148 raft_consensus.cc:493] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:34.529687 15148 raft_consensus.cc:3060] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:34.536636 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet a0f9c321e673433498a9059dd2e2efe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:34.536799 14904 ts_tablet_manager.cc:1916] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:34.537576 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 5daba165262e41af8d3481d3b4929092 with delete_type TABLET_DATA_DELETED (replaced by tablet da52cace2aab4f9dacef18ec5fa56aaf) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:34.537613 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a6e8f55c1ec74e60b16628cac13435f5" } new_table_name: "default.Table_71b84842b7c84927ad89c5ad951be1a1" modify_external_catalogs: true
I20251211 08:57:34.539175 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet d235bade012540e1bb1c0dd871eff1bb (table default.Table_7c70be2e41de4127a511a479fdb6ad7e [id=5b6d7c4fd998498f91fe1594382caed9]) successfully deleted
I20251211 08:57:34.533798 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccce5a446ac9460c9c9b7de72d875d95" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:34.540920 14448 raft_consensus.cc:2468] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
W20251211 08:57:34.538326 14385 leader_election.cc:343] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:34.542428 14385 leader_election.cc:343] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:34.543175 14385 leader_election.cc:304] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:34.542963 15152 raft_consensus.cc:493] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:34.543645 15152 raft_consensus.cc:515] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.544226 15122 raft_consensus.cc:2804] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:34.544518 15122 raft_consensus.cc:493] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:34.544816 15122 raft_consensus.cc:3060] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:34.546015 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cd6e4352cfc41e497dc58d158e24ff2" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:34.546507 14448 raft_consensus.cc:2468] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:34.547595 14542 leader_election.cc:304] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:34.548291 15089 raft_consensus.cc:2804] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:34.548625 15089 raft_consensus.cc:493] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:34.548923 15089 raft_consensus.cc:3060] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:34.552263 15122 raft_consensus.cc:515] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:34.555855 14382 leader_election.cc:343] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 60b23163906447c3877db631968f5623
I20251211 08:57:34.555549 15089 raft_consensus.cc:515] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.547617 15161 consensus_queue.cc:1048] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20251211 08:57:34.556783 15122 leader_election.cc:290] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:34.558418 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cd6e4352cfc41e497dc58d158e24ff2" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:34.558935 14448 raft_consensus.cc:3060] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:34.530627 15030 raft_consensus.cc:359] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.561106 15030 raft_consensus.cc:385] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.561338 15030 raft_consensus.cc:740] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.561932 15030 consensus_queue.cc:260] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.562448 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_c2b453de9d7c46ed92713e510df27dca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:34.496603 14714 consensus_queue.cc:260] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:34.563838 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2b453de9d7c46ed92713e510df27dca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:34.565426 14448 raft_consensus.cc:2468] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:34.566365 14542 leader_election.cc:304] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:34.577173 15152 leader_election.cc:290] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:34.586588 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_26f4e336df6947149cb311c15bee56c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:34.587550 14231 consensus_peers.cc:597] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b9b338a3f544ccb813247250b0f0eb7. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:34.588194 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26f4e336df6947149cb311c15bee56c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:34.593801 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60b23163906447c3877db631968f5623" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:34.594331 14294 raft_consensus.cc:3060] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:34.594945 15089 leader_election.cc:290] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:34.592468 15152 raft_consensus.cc:2804] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:34.601184 14294 raft_consensus.cc:2468] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:34.605581 14385 leader_election.cc:304] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
W20251211 08:57:34.606103 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 101445us
I20251211 08:57:34.596216 15152 raft_consensus.cc:697] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:34.606527 15122 raft_consensus.cc:2804] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:34.607219 15122 raft_consensus.cc:697] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:34.608964 15122 consensus_queue.cc:237] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.564802 14714 ts_tablet_manager.cc:1434] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.098s user 0.004s sys 0.000s
I20251211 08:57:34.610790 15152 consensus_queue.cc:237] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:34.614344 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 48079.9us
I20251211 08:57:34.615017 14714 tablet_bootstrap.cc:492] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:34.617733 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cd6e4352cfc41e497dc58d158e24ff2" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:34.618614 15156 log.cc:927] Time spent T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20251211 08:57:34.618897 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cd6e4352cfc41e497dc58d158e24ff2" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:34.619962 14544 leader_election.cc:343] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:34.621052 14544 leader_election.cc:343] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:34.623925 15030 ts_tablet_manager.cc:1434] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.095s user 0.006s sys 0.000s
I20251211 08:57:34.624768 15030 tablet_bootstrap.cc:492] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:34.628016 14905 ts_tablet_manager.cc:1929] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:34.628363 14905 log.cc:1199] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/292b641df22f478d8849ff6148ecf951
I20251211 08:57:34.629026 14905 ts_tablet_manager.cc:1950] T 292b641df22f478d8849ff6148ecf951 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:34.633989 15030 tablet_bootstrap.cc:654] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.618484 15160 consensus_queue.cc:1048] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:34.643579 14722 tablet_bootstrap.cc:654] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.661522 15148 raft_consensus.cc:515] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.665808 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 106) took 1808 ms. Trace:
I20251211 08:57:34.666107 14905 rpcz_store.cc:276] 1211 08:57:32.857081 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.857184 (+ 103us) service_pool.cc:225] Handling call
1211 08:57:34.505676 (+1648492us) ts_tablet_manager.cc:1151] Deleting tablet 292b641df22f478d8849ff6148ecf951
1211 08:57:34.505708 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.577238 (+ 71530us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.627830 (+ 50592us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.665751 (+ 37921us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1648015}
I20251211 08:57:34.667150 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 292b641df22f478d8849ff6148ecf951 (table default.Table_adee98d887af4b9bb8d59ea922bd36f9 [id=1d8b25ddadc64095ad0422142f43812b]) successfully deleted
I20251211 08:57:34.669260 14009 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_81eee4b23e9045848867df8a0f511ae5" } modify_external_catalogs: true
I20251211 08:57:34.669970 14009 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_81eee4b23e9045848867df8a0f511ae5" } modify_external_catalogs: true
I20251211 08:57:34.684348 14714 tablet_bootstrap.cc:654] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.687011 14004 master_service.cc:954] ReplaceTablet: received request to replace tablet a6570281e1064c79b6e2297723f28658 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:34.691970 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_d86a1b65094448658217d68127970c1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:34.692814 14905 tablet_replica.cc:333] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:57:34.693148 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d86a1b65094448658217d68127970c1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:34.693490 14905 raft_consensus.cc:2243] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:34.693905 14905 raft_consensus.cc:2272] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:34.700397 14905 ts_tablet_manager.cc:1916] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:34.716187 14714 tablet_bootstrap.cc:492] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:34.721764 14714 ts_tablet_manager.cc:1403] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.111s user 0.009s sys 0.000s
I20251211 08:57:34.723801 14714 raft_consensus.cc:359] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.724344 14714 raft_consensus.cc:385] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.728161 14714 raft_consensus.cc:740] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.728785 14714 consensus_queue.cc:260] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.741762 14722 tablet_bootstrap.cc:492] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:34.742168 14722 ts_tablet_manager.cc:1403] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.247s user 0.005s sys 0.005s
W20251211 08:57:34.743839 14382 consensus_peers.cc:597] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15f7bcd680064b9f8fe51aa05acce6fb. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:34.744575 14722 raft_consensus.cc:359] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.746503 14714 ts_tablet_manager.cc:1434] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.024s user 0.004s sys 0.000s
I20251211 08:57:34.749974 15030 tablet_bootstrap.cc:492] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:34.750380 15030 ts_tablet_manager.cc:1403] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.126s user 0.017s sys 0.002s
I20251211 08:57:34.754302 15030 raft_consensus.cc:359] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.754935 15030 raft_consensus.cc:385] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.755304 15030 raft_consensus.cc:740] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.755951 15030 consensus_queue.cc:260] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.752954 14714 tablet_bootstrap.cc:492] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:34.762053 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 1976d584cc0a450ab809ead258a251e2
I20251211 08:57:34.763141 14714 tablet_bootstrap.cc:654] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.763762 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet 1976d584cc0a450ab809ead258a251e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:34.765115 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 1976d584cc0a450ab809ead258a251e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1976d584cc0a450ab809ead258a251e2 already in progress: creating tablet
I20251211 08:57:34.765736 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 1976d584cc0a450ab809ead258a251e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:34.766783 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet 1976d584cc0a450ab809ead258a251e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:34.768404 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccce5a446ac9460c9c9b7de72d875d95" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:34.768966 14446 raft_consensus.cc:3060] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:34.768163 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_150512d0e0b5469694652b5e2e39dabc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:34.770501 14385 consensus_peers.cc:597] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15f7bcd680064b9f8fe51aa05acce6fb. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:34.770628 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_150512d0e0b5469694652b5e2e39dabc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:34.775928 14446 raft_consensus.cc:2468] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:34.776860 14231 leader_election.cc:304] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:34.777673 15158 raft_consensus.cc:2804] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:34.778105 15158 raft_consensus.cc:697] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:34.778906 15158 consensus_queue.cc:237] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.779856 15030 ts_tablet_manager.cc:1434] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.029s user 0.003s sys 0.004s
I20251211 08:57:34.781517 15030 tablet_bootstrap.cc:492] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:34.788532 15148 leader_election.cc:290] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:34.790956 14009 master_service.cc:954] ReplaceTablet: received request to replace tablet e961c5a3b5f640cbaea78b6185048d04 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:34.794023 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2f325f51dd9f433f98cfef7169cdbe0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:34.797197 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f325f51dd9f433f98cfef7169cdbe0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:34.745177 14722 raft_consensus.cc:385] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.801978 14905 ts_tablet_manager.cc:1929] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:34.798691 14722 raft_consensus.cc:740] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.808964 14722 consensus_queue.cc:260] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.812019 14714 tablet_bootstrap.cc:492] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:34.812387 14714 ts_tablet_manager.cc:1403] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.000s
I20251211 08:57:34.814374 14714 raft_consensus.cc:359] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.815043 14714 raft_consensus.cc:385] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.815248 14714 raft_consensus.cc:740] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.814153 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet a6570281e1064c79b6e2297723f28658
I20251211 08:57:34.815698 14714 consensus_queue.cc:260] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.816602 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet a6570281e1064c79b6e2297723f28658 with delete_type TABLET_DATA_DELETED (replaced by tablet e9905227c0a844d199362a442e7d630c) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:34.817025 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet a6570281e1064c79b6e2297723f28658 with delete_type TABLET_DATA_DELETED (replaced by tablet e9905227c0a844d199362a442e7d630c) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:34.818274 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet a6570281e1064c79b6e2297723f28658 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6570281e1064c79b6e2297723f28658 already in progress: creating tablet
I20251211 08:57:34.820040 14722 ts_tablet_manager.cc:1434] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.078s user 0.006s sys 0.000s
I20251211 08:57:34.820551 14004 catalog_manager.cc:6724] ReplaceTablet: tablet a6570281e1064c79b6e2297723f28658 deleted and replaced by tablet e9905227c0a844d199362a442e7d630c
I20251211 08:57:34.821470 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_73091d56834e4e78b870257988c87c57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:34.823729 14722 tablet_bootstrap.cc:492] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:34.833300 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73091d56834e4e78b870257988c87c57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:34.838276 14714 ts_tablet_manager.cc:1434] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20251211 08:57:34.842824 14714 tablet_bootstrap.cc:492] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:34.806599 14904 ts_tablet_manager.cc:1929] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:34.843325 14904 log.cc:1199] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/e015d0fb9d6c4c80a510c05244b334e4
I20251211 08:57:34.844075 14904 ts_tablet_manager.cc:1950] T e015d0fb9d6c4c80a510c05244b334e4 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:34.844710 14011 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 6de43c843ff845a0bd4eb98b6054dd87 (table default.Table_3c7e457251e74eb7b8415bbcb6b6d7db [id=e5ffdeadfa904d85b68d0925da7e7a4c]). Expected version 1 got 0
I20251211 08:57:34.845548 14011 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 5cf1210733754f43885a7c0a350aa206 (table default.Table_2d7127602bd94e2dbc13d3cfae809a87 [id=e9973d6c9b1341d1b96f04367e44e212]). Expected version 2 got 0
I20251211 08:57:34.846087 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 112) took 1782 ms. Trace:
I20251211 08:57:34.846171 14011 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 3303dc521d694be6b386a35da1172ede (table default.Table_a6e8f55c1ec74e60b16628cac13435f5 [id=ebfd6e4105a8410bb79a6db704ec0ba5]). Expected version 1 got 0
I20251211 08:57:34.856658 14714 tablet_bootstrap.cc:654] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.856698 14904 rpcz_store.cc:276] 1211 08:57:33.063992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.064078 (+ 86us) service_pool.cc:225] Handling call
1211 08:57:34.501398 (+1437320us) ts_tablet_manager.cc:1151] Deleting tablet e015d0fb9d6c4c80a510c05244b334e4
1211 08:57:34.501459 (+ 61us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.749587 (+248128us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.806434 (+ 56847us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.846030 (+ 39596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1436864}
I20251211 08:57:34.861248 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet e015d0fb9d6c4c80a510c05244b334e4 (table default.Table_f647a2c75e7c46939a62573dea122fef [id=779be79e440a45d7ac4f6afb05bed936]) successfully deleted
I20251211 08:57:34.862152 14904 tablet_replica.cc:333] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:34.863970 14904 raft_consensus.cc:2243] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:34.864823 14904 raft_consensus.cc:2272] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:34.850368 14722 tablet_bootstrap.cc:654] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.866724 14904 ts_tablet_manager.cc:1916] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:34.877329 14011 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet eb5a6e533e1f4da7b03a800d029f712d (table default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85]). Expected version 1 got 0
W20251211 08:57:34.878386 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3303dc521d694be6b386a35da1172ede consensus state has no leader
I20251211 08:57:34.884759 14271 tablet_service.cc:1505] Processing CreateTablet for tablet dca342fd9de241b5884600199f06fbeb (DEFAULT_TABLE table=default.Table_f627382c623a4f818caf527706e776ab [id=22f3369559b34f5aa54c92c1bb01c2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:34.886873 14429 tablet_service.cc:1505] Processing CreateTablet for tablet dca342fd9de241b5884600199f06fbeb (DEFAULT_TABLE table=default.Table_f627382c623a4f818caf527706e776ab [id=22f3369559b34f5aa54c92c1bb01c2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:34.887885 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dca342fd9de241b5884600199f06fbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:34.891995 14714 tablet_bootstrap.cc:492] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:34.892354 14714 ts_tablet_manager.cc:1403] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.000s
I20251211 08:57:34.894336 14714 raft_consensus.cc:359] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.899057 14714 raft_consensus.cc:385] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.899283 14714 raft_consensus.cc:740] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.899888 14714 consensus_queue.cc:260] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.906226 14714 ts_tablet_manager.cc:1434] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20251211 08:57:34.907006 14714 tablet_bootstrap.cc:492] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:34.912459 14012 catalog_manager.cc:5654] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:34.914167 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 5cf1210733754f43885a7c0a350aa206 (table default.Table_2d7127602bd94e2dbc13d3cfae809a87 [id=e9973d6c9b1341d1b96f04367e44e212]). Expected version 2 got 0
I20251211 08:57:34.916003 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 6de43c843ff845a0bd4eb98b6054dd87 (table default.Table_3c7e457251e74eb7b8415bbcb6b6d7db [id=e5ffdeadfa904d85b68d0925da7e7a4c]). Expected version 1 got 0
I20251211 08:57:34.804055 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccce5a446ac9460c9c9b7de72d875d95" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:34.922942 14905 log.cc:1199] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/426e26859b9c4966b8ec8ba3eab53a43
I20251211 08:57:34.923044 14604 raft_consensus.cc:3060] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:34.923624 14905 ts_tablet_manager.cc:1950] T 426e26859b9c4966b8ec8ba3eab53a43 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:34.925293 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 105) took 2068 ms. Trace:
I20251211 08:57:34.925541 14905 rpcz_store.cc:276] 1211 08:57:32.856778 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:32.878834 (+ 22056us) service_pool.cc:225] Handling call
1211 08:57:34.692677 (+1813843us) ts_tablet_manager.cc:1151] Deleting tablet 426e26859b9c4966b8ec8ba3eab53a43
1211 08:57:34.692713 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.740984 (+ 48271us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.801766 (+ 60782us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.925242 (+123476us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1813259}
I20251211 08:57:34.820161 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet a6570281e1064c79b6e2297723f28658 with delete_type TABLET_DATA_DELETED (replaced by tablet e9905227c0a844d199362a442e7d630c) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:34.926964 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 426e26859b9c4966b8ec8ba3eab53a43 (table default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6]) successfully deleted
I20251211 08:57:34.831336 15030 tablet_bootstrap.cc:654] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.930610 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dca342fd9de241b5884600199f06fbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:34.846064 15152 raft_consensus.cc:493] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:34.931111 15152 raft_consensus.cc:515] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.933537 15152 leader_election.cc:290] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:34.888751 14577 tablet_service.cc:1505] Processing CreateTablet for tablet dca342fd9de241b5884600199f06fbeb (DEFAULT_TABLE table=default.Table_f627382c623a4f818caf527706e776ab [id=22f3369559b34f5aa54c92c1bb01c2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:34.934572 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dca342fd9de241b5884600199f06fbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:34.905454 15163 raft_consensus.cc:493] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:34.935580 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_9d16e03fa298473cb4acdb00f6143d5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:34.935600 15163 raft_consensus.cc:515] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.936084 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 578404732eef4e51b66b6aebffa87cdb (DEFAULT_TABLE table=default.Table_10e56e6be8ad4d798a3d979bbb66ffc3 [id=26c0a9d9b1f84372b18a54f045eb768c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:34.937013 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d16e03fa298473cb4acdb00f6143d5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:34.937162 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578404732eef4e51b66b6aebffa87cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:34.937312 15163 leader_election.cc:290] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:34.938830 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e307e4d1994af0b99a88b3b06b73a4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:34.938871 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e307e4d1994af0b99a88b3b06b73a4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:34.939098 14722 tablet_bootstrap.cc:492] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:34.939509 14722 ts_tablet_manager.cc:1403] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.119s user 0.007s sys 0.003s
W20251211 08:57:34.940487 14544 leader_election.cc:343] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:34.941823 14722 raft_consensus.cc:359] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:34.942164 14542 leader_election.cc:343] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:34.942387 14722 raft_consensus.cc:385] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:34.942544 14542 leader_election.cc:304] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:34.944394 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afe339c4cc95490683618e18a3da252b" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:34.945562 14904 ts_tablet_manager.cc:1929] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:34.945955 14904 log.cc:1199] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/7b9b338a3f544ccb813247250b0f0eb7
I20251211 08:57:34.942620 14722 raft_consensus.cc:740] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:34.947216 14722 consensus_queue.cc:260] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.949059 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 578404732eef4e51b66b6aebffa87cdb (DEFAULT_TABLE table=default.Table_10e56e6be8ad4d798a3d979bbb66ffc3 [id=26c0a9d9b1f84372b18a54f045eb768c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:34.949980 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578404732eef4e51b66b6aebffa87cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:34.949956 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 578404732eef4e51b66b6aebffa87cdb (DEFAULT_TABLE table=default.Table_10e56e6be8ad4d798a3d979bbb66ffc3 [id=26c0a9d9b1f84372b18a54f045eb768c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:34.950883 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578404732eef4e51b66b6aebffa87cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:34.950798 14272 tablet_service.cc:1505] Processing CreateTablet for tablet c53035e83710403bac3a2b401c8c97ad (DEFAULT_TABLE table=default.Table_5a97657fb7304e73bddac0c22b7ee787 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:34.951787 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c53035e83710403bac3a2b401c8c97ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:34.953624 14722 ts_tablet_manager.cc:1434] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20251211 08:57:34.954442 14722 tablet_bootstrap.cc:492] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:34.955945 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afe339c4cc95490683618e18a3da252b" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:34.946626 14904 ts_tablet_manager.cc:1950] T 7b9b338a3f544ccb813247250b0f0eb7 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:34.964463 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 114) took 1871 ms. Trace:
I20251211 08:57:34.964793 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet f909d82e0136470a810e1063d6769868
I20251211 08:57:34.964744 14904 rpcz_store.cc:276] 1211 08:57:33.093394 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.093504 (+ 110us) service_pool.cc:225] Handling call
1211 08:57:34.862001 (+1768497us) ts_tablet_manager.cc:1151] Deleting tablet 7b9b338a3f544ccb813247250b0f0eb7
1211 08:57:34.862053 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.889593 (+ 27540us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.945383 (+ 55790us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:34.964413 (+ 19030us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1767957}
W20251211 08:57:34.965833 14544 leader_election.cc:343] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:34.966007 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 121) took 1261 ms. Trace:
W20251211 08:57:34.966435 14542 leader_election.cc:343] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:34.966499 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet f909d82e0136470a810e1063d6769868 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:34.966805 14542 leader_election.cc:304] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
W20251211 08:57:34.967880 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f909d82e0136470a810e1063d6769868 because the tablet was not found. No further retry: Not found: Tablet not found: f909d82e0136470a810e1063d6769868
I20251211 08:57:34.969272 15163 raft_consensus.cc:493] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:34.969671 15163 raft_consensus.cc:515] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:34.970677 15162 raft_consensus.cc:2749] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:34.971215 15162 raft_consensus.cc:2749] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:34.972338 15163 leader_election.cc:290] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:34.973579 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1976d584cc0a450ab809ead258a251e2" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:34.974346 14722 tablet_bootstrap.cc:654] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.974742 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet f909d82e0136470a810e1063d6769868 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:34.974759 14544 leader_election.cc:343] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:34.975337 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 7b9b338a3f544ccb813247250b0f0eb7 (table default.Table_f0b7cb6abf9b49fd8b0051bb2eb9e55e [id=85b5244918f44d04a067fe1aa8b35975]) successfully deleted
I20251211 08:57:34.976572 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet f909d82e0136470a810e1063d6769868 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:34.976897 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1976d584cc0a450ab809ead258a251e2" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:34.980746 14604 raft_consensus.cc:2468] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:34.989140 14714 tablet_bootstrap.cc:654] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:34.966264 14904 rpcz_store.cc:276] 1211 08:57:33.704757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.706845 (+ 2088us) service_pool.cc:225] Handling call
1211 08:57:34.965661 (+1258816us) ts_tablet_manager.cc:1151] Deleting tablet a4e307e4d1994af0b99a88b3b06b73a4
1211 08:57:34.965717 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.965962 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1258205}
I20251211 08:57:34.991319 14904 tablet_replica.cc:333] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b: stopping tablet replica
W20251211 08:57:34.991618 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet a4e307e4d1994af0b99a88b3b06b73a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4e307e4d1994af0b99a88b3b06b73a4 already in progress: creating tablet
I20251211 08:57:35.003742 14424 tablet_service.cc:1505] Processing CreateTablet for tablet c53035e83710403bac3a2b401c8c97ad (DEFAULT_TABLE table=default.Table_5a97657fb7304e73bddac0c22b7ee787 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:34.981361 14542 leader_election.cc:343] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:35.004524 14542 leader_election.cc:304] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:35.004765 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c53035e83710403bac3a2b401c8c97ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.005127 14584 tablet_service.cc:1505] Processing CreateTablet for tablet c53035e83710403bac3a2b401c8c97ad (DEFAULT_TABLE table=default.Table_5a97657fb7304e73bddac0c22b7ee787 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.005468 15122 raft_consensus.cc:493] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:35.005493 15152 raft_consensus.cc:2749] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:35.006044 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c53035e83710403bac3a2b401c8c97ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.005841 15122 raft_consensus.cc:515] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.006649 14904 raft_consensus.cc:2243] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:35.007284 14904 raft_consensus.cc:2272] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:35.007736 15122 leader_election.cc:290] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.009658 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9168007ca0584954a734e2c8f2e73f70" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:35.010293 14294 raft_consensus.cc:2468] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:35.011301 14904 ts_tablet_manager.cc:1916] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:35.011948 14385 leader_election.cc:304] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:35.012650 15122 raft_consensus.cc:2804] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:35.012979 15122 raft_consensus.cc:493] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:35.013262 15122 raft_consensus.cc:3060] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:34.926615 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 108) took 1884 ms. Trace:
I20251211 08:57:35.016198 14905 rpcz_store.cc:276] 1211 08:57:33.041964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.042855 (+ 891us) service_pool.cc:225] Handling call
1211 08:57:34.926377 (+1883522us) ts_tablet_manager.cc:1151] Deleting tablet d2c6080b3c1e4897bc1881e50f29f3a0
1211 08:57:34.926409 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:34.926558 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1881078}
W20251211 08:57:35.018890 14385 consensus_peers.cc:597] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:35.018633 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ea1935de13854b0b82e1382effe6cfbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:35.020011 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea1935de13854b0b82e1382effe6cfbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:35.024911 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_6cbe43becdc846bcab4b1233f65cda0c" } modify_external_catalogs: true
I20251211 08:57:35.025349 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_6cbe43becdc846bcab4b1233f65cda0c" } modify_external_catalogs: true
I20251211 08:57:35.030164 15122 raft_consensus.cc:515] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.041380 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9168007ca0584954a734e2c8f2e73f70" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:35.041893 14294 raft_consensus.cc:3060] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.048394 14294 raft_consensus.cc:2468] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:35.026091 15030 tablet_bootstrap.cc:492] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:35.051347 15030 ts_tablet_manager.cc:1403] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.270s user 0.015s sys 0.000s
I20251211 08:57:35.051304 14385 leader_election.cc:304] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
W20251211 08:57:35.051764 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet d2c6080b3c1e4897bc1881e50f29f3a0 because the tablet was not found. No further retry: Not found: Tablet not found: d2c6080b3c1e4897bc1881e50f29f3a0
I20251211 08:57:35.052066 15083 raft_consensus.cc:2804] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:35.053514 15030 raft_consensus.cc:359] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.054088 15030 raft_consensus.cc:385] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:35.054416 15030 raft_consensus.cc:740] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:35.055634 15030 consensus_queue.cc:260] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.058522 15122 leader_election.cc:290] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.061086 14722 tablet_bootstrap.cc:492] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:35.062408 14722 ts_tablet_manager.cc:1403] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.108s user 0.009s sys 0.000s
I20251211 08:57:35.062009 14006 catalog_manager.cc:5654] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:35.064242 14013 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_84c8a91dc6244ac0b8dd3a2986479da6 [id=3f3f000cd93e44df9f084873ba4b168b] alter complete (version 1)
I20251211 08:57:35.065012 14722 raft_consensus.cc:359] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.065531 14722 raft_consensus.cc:385] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:35.065780 14722 raft_consensus.cc:740] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
W20251211 08:57:35.065960 14013 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 24) took 1477 ms (client timeout 1000 ms). Trace:
I20251211 08:57:35.066819 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet e015d0fb9d6c4c80a510c05244b334e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:29 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:35.068186 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet e015d0fb9d6c4c80a510c05244b334e4 because the tablet was not found. No further retry: Not found: Tablet not found: e015d0fb9d6c4c80a510c05244b334e4
I20251211 08:57:35.031160 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9168007ca0584954a734e2c8f2e73f70" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:35.069290 14008 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55614 (request call id 189) took 1089 ms. Trace:
I20251211 08:57:35.050181 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9168007ca0584954a734e2c8f2e73f70" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:35.069965 14606 raft_consensus.cc:3060] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:35.066231 14013 rpcz_store.cc:269] 1211 08:57:33.588959 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.707191 (+118232us) service_pool.cc:225] Handling call
1211 08:57:33.711347 (+ 4156us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:35.065904 (+1354557us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:33.978153 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:33.978490 (+ 337us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:33.978512 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:57:33.978524 (+ 12us) tablet.cc:660] Decoding operations
1211 08:57:33.978747 (+ 223us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:33.978801 (+ 54us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:33.978821 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:33.978947 (+ 126us) tablet.cc:699] Row locks acquired
1211 08:57:33.978962 (+ 15us) write_op.cc:260] PREPARE: finished
1211 08:57:33.979054 (+ 92us) write_op.cc:270] Start()
1211 08:57:33.979141 (+ 87us) write_op.cc:276] Timestamp: P: 1765443453979201 usec, L: 0
1211 08:57:33.979156 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:57:33.979556 (+ 400us) log.cc:844] Serialized 356 byte log entry
1211 08:57:34.063267 (+ 83711us) op_driver.cc:464] REPLICATION: finished
1211 08:57:34.064668 (+ 1401us) write_op.cc:301] APPLY: starting
1211 08:57:34.064719 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:34.064800 (+ 81us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:34.064894 (+ 94us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:34.065025 (+ 131us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:34.065158 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:34.065200 (+ 42us) write_op.cc:312] APPLY: finished
1211 08:57:34.065369 (+ 169us) log.cc:844] Serialized 23 byte log entry
1211 08:57:34.065495 (+ 126us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:34.065551 (+ 56us) write_op.cc:454] Released schema lock
1211 08:57:34.065586 (+ 35us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:34.065817 (+ 231us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:34.065839 (+ 22us) write_op.cc:454] Released schema lock
Related trace 'op':
1211 08:57:34.280416 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:34.280827 (+ 411us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:34.280871 (+ 44us) write_op.cc:435] Acquired schema lock
1211 08:57:34.280885 (+ 14us) tablet.cc:660] Decoding operations
1211 08:57:34.281146 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:34.281186 (+ 40us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:34.281214 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:34.281358 (+ 144us) tablet.cc:699] Row locks acquired
1211 08:57:34.281373 (+ 15us) write_op.cc:260] PREPARE: finished
1211 08:57:34.281479 (+ 106us) write_op.cc:270] Start()
1211 08:57:34.281562 (+ 83us) write_op.cc:276] Timestamp: P: 1765443454281542 usec, L: 0
1211 08:57:34.281579 (+ 17us) op_driver.cc:348] REPLICATION: starting
1211 08:57:34.282068 (+ 489us) log.cc:844] Serialized 356 byte log entry
1211 08:57:34.319989 (+ 37921us) op_driver.cc:464] REPLICATION: finished
1211 08:57:34.321849 (+ 1860us) write_op.cc:301] APPLY: starting
1211 08:57:34.321892 (+ 43us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:34.321968 (+ 76us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:34.322068 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:34.322173 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:34.322265 (+ 92us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:34.322307 (+ 42us) write_op.cc:312] APPLY: finished
1211 08:57:34.322417 (+ 110us) log.cc:844] Serialized 23 byte log entry
1211 08:57:34.322506 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:34.322549 (+ 43us) write_op.cc:454] Released schema lock
1211 08:57:34.322571 (+ 22us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:34.325269 (+ 2698us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:34.325291 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9964,"reported_tablets":5,"spinlock_wait_cycles":3866880,"child_traces":[["op",{"apply.queue_time_us":1278,"apply.run_cpu_time_us":1116,"apply.run_wall_time_us":1115,"num_ops":1,"prepare.queue_time_us":181583,"prepare.run_cpu_time_us":1682,"prepare.run_wall_time_us":15668,"raft.queue_time_us":347,"raft.run_cpu_time_us":475,"raft.run_wall_time_us":717,"replication_time_us":84006}],["op",{"apply.queue_time_us":1747,"apply.run_cpu_time_us":882,"apply.run_wall_time_us":881,"num_ops":1,"prepare.queue_time_us":211403,"prepare.run_cpu_time_us":1974,"prepare.run_wall_time_us":44860,"raft.queue_time_us":100,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":166,"replication_time_us":38342,"thread_start_us":4233,"threads_started":1}]]}
I20251211 08:57:35.072705 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet db302282bc3d4d8cb56d1cbb2f73c4fe from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:35.073261 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b9b338a3f544ccb813247250b0f0eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:35.074465 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 7b9b338a3f544ccb813247250b0f0eb7 because the tablet was not found. No further retry: Not found: Tablet not found: 7b9b338a3f544ccb813247250b0f0eb7
I20251211 08:57:35.076809 15030 ts_tablet_manager.cc:1434] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20251211 08:57:35.077085 14714 tablet_bootstrap.cc:492] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
W20251211 08:57:35.077152 14010 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 35) took 1252 ms (client timeout 1000 ms). Trace:
I20251211 08:57:35.077456 14714 ts_tablet_manager.cc:1403] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.171s user 0.004s sys 0.004s
I20251211 08:57:35.079771 14714 raft_consensus.cc:359] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.081254 14714 raft_consensus.cc:385] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:35.081507 14714 raft_consensus.cc:740] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:35.082135 14714 consensus_queue.cc:260] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.083009 14577 tablet_service.cc:1552] Processing DeleteTablet for tablet a9524aa1503549bd820918d1c025aa2e with delete_type TABLET_DATA_DELETED (replaced by tablet db302282bc3d4d8cb56d1cbb2f73c4fe) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.083611 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet 15f7bcd680064b9f8fe51aa05acce6fb with delete_type TABLET_DATA_DELETED (replaced by tablet d4f826d079774152a3b66bfb81f990a8) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.069667 14008 rpcz_store.cc:276] 1211 08:57:33.979763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.980441 (+ 678us) service_pool.cc:225] Handling call
1211 08:57:33.995391 (+ 14950us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da87016 00007fc20b6b5b24 00007fc20b6b5ab2 00007fc20b6b59e1 00007fc20b6b595c 00007fc20b6b4c12 00007fc20998455c 00007fc209983996
1211 08:57:33.995897 (+ 506us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:33.996410 (+ 513us) catalog_manager.cc:3708] Apply alter schema
1211 08:57:33.996892 (+ 482us) catalog_manager.cc:3740] Acquired catalog manager lock
1211 08:57:33.997112 (+ 220us) catalog_manager.cc:3898] Updating metadata on disk
1211 08:57:34.323997 (+326885us) catalog_manager.cc:3929] Committing alterations to in-memory state
1211 08:57:35.069240 (+745243us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:34.198446 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:34.204389 (+ 5943us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:34.204419 (+ 30us) write_op.cc:435] Acquired schema lock
1211 08:57:34.204429 (+ 10us) tablet.cc:660] Decoding operations
1211 08:57:34.204671 (+ 242us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:34.204701 (+ 30us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:34.204722 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:34.204847 (+ 125us) tablet.cc:699] Row locks acquired
1211 08:57:34.204858 (+ 11us) write_op.cc:260] PREPARE: finished
1211 08:57:34.205686 (+ 828us) write_op.cc:270] Start()
1211 08:57:34.205762 (+ 76us) write_op.cc:276] Timestamp: P: 1765443454204978 usec, L: 0
1211 08:57:34.205783 (+ 21us) op_driver.cc:348] REPLICATION: starting
1211 08:57:34.206174 (+ 391us) log.cc:844] Serialized 405 byte log entry
1211 08:57:34.318966 (+112792us) op_driver.cc:464] REPLICATION: finished
1211 08:57:34.320331 (+ 1365us) write_op.cc:301] APPLY: starting
1211 08:57:34.320382 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:34.320454 (+ 72us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:34.320542 (+ 88us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:34.320661 (+ 119us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:34.320788 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:34.320850 (+ 62us) write_op.cc:312] APPLY: finished
1211 08:57:34.321058 (+ 208us) log.cc:844] Serialized 23 byte log entry
1211 08:57:34.321227 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:34.321301 (+ 74us) write_op.cc:454] Released schema lock
1211 08:57:34.321341 (+ 40us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:34.321556 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:34.321592 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":32723712,"child_traces":[["op",{"apply.queue_time_us":1252,"apply.run_cpu_time_us":1193,"apply.run_wall_time_us":1192,"num_ops":1,"prepare.queue_time_us":200369,"prepare.run_cpu_time_us":1613,"prepare.run_wall_time_us":19033,"replication_time_us":113106}]]}
I20251211 08:57:35.085409 14008 master_service.cc:954] ReplaceTablet: received request to replace tablet e961c5a3b5f640cbaea78b6185048d04 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:35.085974 14581 tablet_service.cc:1552] Processing DeleteTablet for tablet 25805a95349d49f58280a30e6876929d with delete_type TABLET_DATA_DELETED (replaced by tablet 41d6ef975a47489385791065bb8f9f46) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.086519 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 1853070f81c34b52b8c74a7402e6dad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:30 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.087527 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 292b641df22f478d8849ff6148ecf951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:32 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:35.077426 14010 rpcz_store.cc:269] 1211 08:57:33.824725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.824833 (+ 108us) service_pool.cc:225] Handling call
1211 08:57:33.844362 (+ 19529us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:35.077098 (+1232736us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:33.993905 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:33.994267 (+ 362us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:33.994288 (+ 21us) write_op.cc:435] Acquired schema lock
1211 08:57:33.994300 (+ 12us) tablet.cc:660] Decoding operations
1211 08:57:33.994512 (+ 212us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:33.994541 (+ 29us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:33.994574 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:33.994687 (+ 113us) tablet.cc:699] Row locks acquired
1211 08:57:33.994698 (+ 11us) write_op.cc:260] PREPARE: finished
1211 08:57:34.003380 (+ 8682us) write_op.cc:270] Start()
1211 08:57:34.003453 (+ 73us) write_op.cc:276] Timestamp: P: 1765443454003534 usec, L: 0
1211 08:57:34.003467 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:34.005250 (+ 1783us) log.cc:844] Serialized 530 byte log entry
1211 08:57:34.064214 (+ 58964us) op_driver.cc:464] REPLICATION: finished
1211 08:57:34.066044 (+ 1830us) write_op.cc:301] APPLY: starting
1211 08:57:34.066080 (+ 36us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:34.066152 (+ 72us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:34.066224 (+ 72us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:34.066333 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:34.066442 (+ 109us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:34.066475 (+ 33us) write_op.cc:312] APPLY: finished
1211 08:57:34.066590 (+ 115us) log.cc:844] Serialized 23 byte log entry
1211 08:57:34.066685 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:34.066727 (+ 42us) write_op.cc:454] Released schema lock
1211 08:57:34.066752 (+ 25us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:34.097100 (+ 30348us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:34.097124 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"spinlock_wait_cycles":40528128,"child_traces":[["op",{"apply.queue_time_us":1721,"apply.run_cpu_time_us":873,"apply.run_wall_time_us":873,"num_ops":1,"prepare.queue_time_us":75672,"prepare.run_cpu_time_us":1722,"prepare.run_wall_time_us":103169,"raft.queue_time_us":94,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":291,"replication_time_us":60678}]]}
I20251211 08:57:35.092397 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet d235bade012540e1bb1c0dd871eff1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:28 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:35.094676 14007 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 26) took 1863 ms (client timeout 1000 ms). Trace:
W20251211 08:57:35.096416 14007 rpcz_store.cc:269] 1211 08:57:33.231576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.232313 (+ 737us) service_pool.cc:225] Handling call
1211 08:57:35.094619 (+1862306us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":22}
I20251211 08:57:35.097113 15148 raft_consensus.cc:493] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:35.097383 14722 consensus_queue.cc:260] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.097510 15148 raft_consensus.cc:515] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.101316 15148 leader_election.cc:290] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.103173 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 (DEFAULT_TABLE table=default.Table_ec0af779c90941db9c7bf85ccff7499f [id=99243463ed7545ff80f928e6664a3748]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.104168 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5a4b6b1b28406c82e2acd9356b60d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.104848 14722 ts_tablet_manager.cc:1434] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.042s user 0.001s sys 0.003s
I20251211 08:57:35.106209 14722 tablet_bootstrap.cc:492] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:35.118387 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e410da32b3446edbf84b2eaa96424c3" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:35.119133 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_885d3fb0ab5f4d0a82972d94a34c2a88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:35.118944 14608 raft_consensus.cc:2468] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
W20251211 08:57:35.120540 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_885d3fb0ab5f4d0a82972d94a34c2a88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:35.120843 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 60b23163906447c3877db631968f5623 with delete_type TABLET_DATA_DELETED (replaced by tablet e961c5a3b5f640cbaea78b6185048d04) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.121084 14010 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_fe86659b8ddb426ab2ba622d58daab57" } new_table_name: "default.Table_90208748f83c43e9b2bf122c5425257e" modify_external_catalogs: true
I20251211 08:57:35.121402 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 426e26859b9c4966b8ec8ba3eab53a43 with delete_type TABLET_DATA_DELETED (replaced by tablet e15483f539f24f1ba1b442663d66fd45) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.125406 14230 leader_election.cc:304] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:35.126510 15148 raft_consensus.cc:2804] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:35.126871 15148 raft_consensus.cc:493] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:35.097019 14714 ts_tablet_manager.cc:1434] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20251211 08:57:35.102684 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e410da32b3446edbf84b2eaa96424c3" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:35.127252 15148 raft_consensus.cc:3060] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.127411 14449 raft_consensus.cc:2468] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:35.127821 14714 tablet_bootstrap.cc:492] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:35.083230 15083 raft_consensus.cc:697] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:35.131106 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b9b338a3f544ccb813247250b0f0eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:35.130731 15083 consensus_queue.cc:237] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.133778 14714 tablet_bootstrap.cc:654] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:35.135478 15030 tablet_bootstrap.cc:492] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:35.137207 15185 tablet.cc:1722] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:35.138481 15183 tablet.cc:1722] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:35.140400 14606 raft_consensus.cc:2468] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:35.141794 15184 tablet.cc:1722] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:35.054090 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 109) took 2005 ms. Trace:
I20251211 08:57:35.147174 14905 rpcz_store.cc:276] 1211 08:57:33.048833 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.053118 (+ 4285us) service_pool.cc:225] Handling call
1211 08:57:35.053873 (+2000755us) ts_tablet_manager.cc:1151] Deleting tablet e015d0fb9d6c4c80a510c05244b334e4
1211 08:57:35.053909 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.054044 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2000232}
W20251211 08:57:35.149106 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet e015d0fb9d6c4c80a510c05244b334e4 because the tablet was not found. No further retry: Not found: Tablet not found: e015d0fb9d6c4c80a510c05244b334e4
I20251211 08:57:35.149415 15030 tablet_bootstrap.cc:654] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:35.151840 14905 tablet_replica.cc:333] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:35.152571 14905 raft_consensus.cc:2243] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:35.153800 14905 raft_consensus.cc:2272] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:35.157326 14905 ts_tablet_manager.cc:1916] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:35.160746 14714 tablet_bootstrap.cc:492] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:35.162089 14714 ts_tablet_manager.cc:1403] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.007s
I20251211 08:57:35.145246 14722 tablet_bootstrap.cc:654] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:35.148118 15148 raft_consensus.cc:515] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.164239 14714 raft_consensus.cc:359] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.164849 14714 raft_consensus.cc:385] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:35.165087 14714 raft_consensus.cc:740] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:35.165032 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e410da32b3446edbf84b2eaa96424c3" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:35.165594 14446 raft_consensus.cc:3060] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.151471 14904 ts_tablet_manager.cc:1929] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:35.166322 14904 log.cc:1199] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/c181adf9db574ab39432b2f9b14fe4d7
I20251211 08:57:35.166841 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e410da32b3446edbf84b2eaa96424c3" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:35.167090 14904 ts_tablet_manager.cc:1950] T c181adf9db574ab39432b2f9b14fe4d7 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:35.167405 14606 raft_consensus.cc:3060] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.168892 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 122) took 1442 ms. Trace:
I20251211 08:57:35.169219 14904 rpcz_store.cc:276] 1211 08:57:33.726941 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.755436 (+ 28495us) service_pool.cc:225] Handling call
1211 08:57:34.991142 (+1235706us) ts_tablet_manager.cc:1151] Deleting tablet c181adf9db574ab39432b2f9b14fe4d7
1211 08:57:34.991211 (+ 69us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.116032 (+124821us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:35.151265 (+ 35233us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:35.168842 (+ 17577us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1227565}
I20251211 08:57:35.170370 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet c181adf9db574ab39432b2f9b14fe4d7 (table default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]) successfully deleted
I20251211 08:57:35.171527 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 123) took 1416 ms. Trace:
I20251211 08:57:35.165642 14714 consensus_queue.cc:260] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.175149 14714 ts_tablet_manager.cc:1434] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20251211 08:57:35.174494 14606 raft_consensus.cc:2468] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:35.175468 14904 rpcz_store.cc:276] 1211 08:57:33.755145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.766877 (+ 11732us) service_pool.cc:225] Handling call
1211 08:57:35.171202 (+1404325us) ts_tablet_manager.cc:1151] Deleting tablet e961c5a3b5f640cbaea78b6185048d04
1211 08:57:35.171248 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.171482 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1361153}
W20251211 08:57:35.176836 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet e961c5a3b5f640cbaea78b6185048d04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e961c5a3b5f640cbaea78b6185048d04 already in progress: creating tablet
I20251211 08:57:35.176359 14230 leader_election.cc:304] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:35.177747 15158 raft_consensus.cc:2804] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:35.177909 14904 tablet_replica.cc:333] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:35.178583 14904 raft_consensus.cc:2243] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:35.180236 15158 raft_consensus.cc:697] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:35.181182 15158 consensus_queue.cc:237] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.182619 14714 tablet_bootstrap.cc:492] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:35.174860 15148 leader_election.cc:290] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.188239 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 (DEFAULT_TABLE table=default.Table_ec0af779c90941db9c7bf85ccff7499f [id=99243463ed7545ff80f928e6664a3748]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.189149 14904 raft_consensus.cc:2272] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:35.189811 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 (DEFAULT_TABLE table=default.Table_ec0af779c90941db9c7bf85ccff7499f [id=99243463ed7545ff80f928e6664a3748]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.190881 14722 tablet_bootstrap.cc:492] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:35.190861 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5a4b6b1b28406c82e2acd9356b60d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.191239 14722 ts_tablet_manager.cc:1403] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.085s user 0.005s sys 0.008s
I20251211 08:57:35.193336 14722 raft_consensus.cc:359] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.194988 15152 raft_consensus.cc:493] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:35.195422 15152 raft_consensus.cc:515] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.189147 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5a4b6b1b28406c82e2acd9356b60d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.197184 15152 leader_election.cc:290] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:35.199393 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a1f12553934446bfe44f4ed41abf9a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:35.201818 14542 leader_election.cc:343] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:35.196826 14904 ts_tablet_manager.cc:1916] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:35.213169 14722 raft_consensus.cc:385] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:35.213413 14722 raft_consensus.cc:740] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:35.216866 14446 raft_consensus.cc:2468] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:35.200418 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a1f12553934446bfe44f4ed41abf9a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:35.219933 14544 leader_election.cc:343] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:35.220257 14544 leader_election.cc:304] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:35.221048 15152 raft_consensus.cc:2749] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:35.222982 14722 consensus_queue.cc:260] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.224854 14722 ts_tablet_manager.cc:1434] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.033s user 0.004s sys 0.000s
I20251211 08:57:35.229190 14722 tablet_bootstrap.cc:492] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:35.230042 13997 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 3303dc521d694be6b386a35da1172ede (table default.Table_71b84842b7c84927ad89c5ad951be1a1 [id=ebfd6e4105a8410bb79a6db704ec0ba5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3303dc521d694be6b386a35da1172ede consensus state has no leader
I20251211 08:57:35.235266 14904 ts_tablet_manager.cc:1929] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:35.235677 14904 log.cc:1199] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/25805a95349d49f58280a30e6876929d
I20251211 08:57:35.236353 14904 ts_tablet_manager.cc:1950] T 25805a95349d49f58280a30e6876929d P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:57:35.236469 14011 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 27) took 989 ms (client timeout 1000 ms). Trace:
W20251211 08:57:35.236766 14011 rpcz_store.cc:269] 1211 08:57:34.246920 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.254870 (+ 7950us) service_pool.cc:225] Handling call
1211 08:57:35.236411 (+981541us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":24}
I20251211 08:57:35.238257 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 128) took 1425 ms. Trace:
I20251211 08:57:35.214680 14714 tablet_bootstrap.cc:654] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:35.241268 14429 tablet_service.cc:1505] Processing CreateTablet for tablet 2016208a33dc4a5cadd7ced8e85ccc65 (DEFAULT_TABLE table=default.Table_face70f737874a76a60d1f4ea105380c [id=f93e9bee7bdd443c9eb3d158f1e40d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.242130 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2016208a33dc4a5cadd7ced8e85ccc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.242093 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 2016208a33dc4a5cadd7ced8e85ccc65 (DEFAULT_TABLE table=default.Table_face70f737874a76a60d1f4ea105380c [id=f93e9bee7bdd443c9eb3d158f1e40d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.243052 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2016208a33dc4a5cadd7ced8e85ccc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.249046 15083 raft_consensus.cc:493] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:35.249481 15083 raft_consensus.cc:515] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.252326 15083 leader_election.cc:290] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.255122 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6de43c843ff845a0bd4eb98b6054dd87" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:35.255633 14606 raft_consensus.cc:2468] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:35.252861 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6de43c843ff845a0bd4eb98b6054dd87" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:35.256564 14382 leader_election.cc:304] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
W20251211 08:57:35.257258 14385 leader_election.cc:343] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:35.257261 15083 raft_consensus.cc:2804] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:35.258193 15083 raft_consensus.cc:493] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:35.258525 15083 raft_consensus.cc:3060] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.260560 14905 ts_tablet_manager.cc:1929] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:35.265373 15083 raft_consensus.cc:515] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.265938 14904 rpcz_store.cc:276] 1211 08:57:33.812710 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.812812 (+ 102us) service_pool.cc:225] Handling call
1211 08:57:35.177769 (+1364957us) ts_tablet_manager.cc:1151] Deleting tablet 25805a95349d49f58280a30e6876929d
1211 08:57:35.177822 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.222127 (+ 44305us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:35.235104 (+ 12977us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:35.238198 (+ 3094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1364411}
I20251211 08:57:35.271250 14905 log.cc:1199] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/7b9b338a3f544ccb813247250b0f0eb7
I20251211 08:57:35.272578 15148 raft_consensus.cc:493] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:35.272949 14905 ts_tablet_manager.cc:1950] T 7b9b338a3f544ccb813247250b0f0eb7 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:35.273087 15148 raft_consensus.cc:515] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.275046 15148 leader_election.cc:290] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.276773 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6e533e1f4da7b03a800d029f712d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:35.277024 14722 tablet_bootstrap.cc:654] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:35.277354 14606 raft_consensus.cc:2468] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:35.277485 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 110) took 2212 ms. Trace:
I20251211 08:57:35.277778 14905 rpcz_store.cc:276] 1211 08:57:33.065146 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.065275 (+ 129us) service_pool.cc:225] Handling call
1211 08:57:35.151698 (+2086423us) ts_tablet_manager.cc:1151] Deleting tablet 7b9b338a3f544ccb813247250b0f0eb7
1211 08:57:35.151748 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.228797 (+ 77049us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:35.260363 (+ 31566us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:35.277436 (+ 17073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2085875}
I20251211 08:57:35.278460 14230 leader_election.cc:304] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:35.279865 15148 raft_consensus.cc:2804] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:35.280295 15148 raft_consensus.cc:493] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:35.280758 15148 raft_consensus.cc:3060] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.287817 14904 tablet_replica.cc:333] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:35.287735 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 7b9b338a3f544ccb813247250b0f0eb7 (table default.Table_f0b7cb6abf9b49fd8b0051bb2eb9e55e [id=85b5244918f44d04a067fe1aa8b35975]) successfully deleted
I20251211 08:57:35.279920 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6e533e1f4da7b03a800d029f712d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:35.288323 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 25805a95349d49f58280a30e6876929d (table default.Table_fa2830322a8a4f7b96cd19543b967b05 [id=52ae4637827d424eb4d37362bf21cea2]) successfully deleted
I20251211 08:57:35.288292 14446 raft_consensus.cc:2468] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:35.288924 15182 raft_consensus.cc:493] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:35.289098 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 111) took 2213 ms. Trace:
I20251211 08:57:35.289295 15182 raft_consensus.cc:515] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.290155 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:31 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.291308 15182 leader_election.cc:290] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.291314 14272 tablet_service.cc:1505] Processing CreateTablet for tablet 2016208a33dc4a5cadd7ced8e85ccc65 (DEFAULT_TABLE table=default.Table_face70f737874a76a60d1f4ea105380c [id=f93e9bee7bdd443c9eb3d158f1e40d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.292239 14904 raft_consensus.cc:2243] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:35.293272 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6de43c843ff845a0bd4eb98b6054dd87" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:35.293781 14600 raft_consensus.cc:3060] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.295578 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6e533e1f4da7b03a800d029f712d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:35.296096 14606 raft_consensus.cc:2468] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:35.297205 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f13b896c35754c0a94cc5b238778ecd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:35.297302 15148 raft_consensus.cc:515] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.297302 14382 leader_election.cc:304] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
W20251211 08:57:35.298646 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f13b896c35754c0a94cc5b238778ecd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:35.291445 15083 leader_election.cc:290] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.299208 14904 raft_consensus.cc:2272] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:35.299373 15122 raft_consensus.cc:2804] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:35.299944 15122 raft_consensus.cc:493] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:35.300244 15122 raft_consensus.cc:3060] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.301285 14600 raft_consensus.cc:2468] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:35.302394 14382 leader_election.cc:304] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:35.302431 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_58593e18b9634b2c8912ff1109e05a52" } modify_external_catalogs: true
I20251211 08:57:35.302912 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_58593e18b9634b2c8912ff1109e05a52" } modify_external_catalogs: true
I20251211 08:57:35.303093 15182 raft_consensus.cc:2804] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:35.289397 14905 rpcz_store.cc:276] 1211 08:57:33.076143 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.076268 (+ 125us) service_pool.cc:225] Handling call
1211 08:57:35.288878 (+2212610us) ts_tablet_manager.cc:1151] Deleting tablet 00f79094ed8b4782a3a1488e41794f9b
1211 08:57:35.288915 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.289050 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2199622}
W20251211 08:57:35.305505 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 00f79094ed8b4782a3a1488e41794f9b because the tablet was not found. No further retry: Not found: Tablet not found: 00f79094ed8b4782a3a1488e41794f9b
I20251211 08:57:35.306421 14904 ts_tablet_manager.cc:1916] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:35.307235 14905 tablet_replica.cc:333] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:35.309870 14905 raft_consensus.cc:2243] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:35.312080 15122 raft_consensus.cc:515] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.313318 14905 raft_consensus.cc:2272] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:35.317731 14714 tablet_bootstrap.cc:492] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:35.318176 14714 ts_tablet_manager.cc:1403] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.136s user 0.011s sys 0.000s
I20251211 08:57:35.319597 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet b8d753358f20476ba23d7d7b72e26e74
I20251211 08:57:35.319857 14905 ts_tablet_manager.cc:1916] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:35.322183 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet b8d753358f20476ba23d7d7b72e26e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.326680 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6e533e1f4da7b03a800d029f712d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:35.327260 14600 raft_consensus.cc:3060] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.331465 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d1d406b278304b5ca6e6cc3b63e7fd36" } new_table_name: "default.Table_4ab19ee3f1f3473fa3d320e7841a201c" modify_external_catalogs: true
I20251211 08:57:35.334300 14600 raft_consensus.cc:2468] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:35.294313 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6e533e1f4da7b03a800d029f712d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:35.335584 14296 raft_consensus.cc:2393] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
I20251211 08:57:35.294991 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6de43c843ff845a0bd4eb98b6054dd87" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:35.314693 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6e533e1f4da7b03a800d029f712d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:35.337092 14298 raft_consensus.cc:2393] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
I20251211 08:57:35.292644 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2016208a33dc4a5cadd7ced8e85ccc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.322461 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet b8d753358f20476ba23d7d7b72e26e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:35.340756 15148 leader_election.cc:290] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.346684 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet b8d753358f20476ba23d7d7b72e26e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:35.347452 14385 leader_election.cc:343] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:35.345669 14230 leader_election.cc:304] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:35.349442 15159 raft_consensus.cc:2804] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:35.349853 15159 raft_consensus.cc:697] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:35.350742 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet 15f7bcd680064b9f8fe51aa05acce6fb with delete_type TABLET_DATA_DELETED (replaced by tablet d4f826d079774152a3b66bfb81f990a8) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:35.350571 15159 consensus_queue.cc:237] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.351333 14718 tablet_replica.cc:333] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:35.351996 14718 raft_consensus.cc:2243] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:35.353832 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet 25805a95349d49f58280a30e6876929d with delete_type TABLET_DATA_DELETED (replaced by tablet 41d6ef975a47489385791065bb8f9f46) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:35.354411 14718 pending_rounds.cc:70] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:35.354635 14718 pending_rounds.cc:77] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256386592927744 op_type: NO_OP noop_request { }
I20251211 08:57:35.354957 14718 raft_consensus.cc:2889] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:35.355233 14718 raft_consensus.cc:2272] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:35.356954 14718 ts_tablet_manager.cc:1916] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:35.363521 14271 tablet_service.cc:1505] Processing CreateTablet for tablet 9ccfcacb5b294a628926d34cd31b137e (DEFAULT_TABLE table=default.Table_93954115dc5d4d2283faa424e83608b0 [id=cb30302a7db04098ae9a5a92ce055d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.363904 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:31 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:35.364513 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ccfcacb5b294a628926d34cd31b137e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.364820 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet 60b23163906447c3877db631968f5623 with delete_type TABLET_DATA_DELETED (replaced by tablet e961c5a3b5f640cbaea78b6185048d04) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:35.366117 15122 leader_election.cc:290] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.366617 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6e533e1f4da7b03a800d029f712d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
W20251211 08:57:35.366972 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet b8d753358f20476ba23d7d7b72e26e74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8d753358f20476ba23d7d7b72e26e74 already in progress: creating tablet
I20251211 08:57:35.367266 14446 raft_consensus.cc:2393] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:35.371898 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6e533e1f4da7b03a800d029f712d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:35.373418 14714 raft_consensus.cc:359] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.374480 14600 raft_consensus.cc:2393] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
I20251211 08:57:35.378266 15192 consensus_queue.cc:1048] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:35.379282 14722 tablet_bootstrap.cc:492] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:35.379418 14714 raft_consensus.cc:385] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:35.379675 14714 raft_consensus.cc:740] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:35.379761 14722 ts_tablet_manager.cc:1403] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.154s user 0.009s sys 0.006s
I20251211 08:57:35.378822 14382 leader_election.cc:304] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:35.380198 14714 consensus_queue.cc:260] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.381414 15192 raft_consensus.cc:2749] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251211 08:57:35.381899 14722 raft_consensus.cc:359] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.382483 14722 raft_consensus.cc:385] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:35.382923 14722 raft_consensus.cc:740] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:35.383651 14722 consensus_queue.cc:260] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.385282 15182 raft_consensus.cc:697] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:35.385941 14722 ts_tablet_manager.cc:1434] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20251211 08:57:35.386068 15182 consensus_queue.cc:237] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.387015 15030 tablet_bootstrap.cc:492] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:35.387395 15030 ts_tablet_manager.cc:1403] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.253s user 0.008s sys 0.026s
I20251211 08:57:35.388181 14429 tablet_service.cc:1505] Processing CreateTablet for tablet 9ccfcacb5b294a628926d34cd31b137e (DEFAULT_TABLE table=default.Table_93954115dc5d4d2283faa424e83608b0 [id=cb30302a7db04098ae9a5a92ce055d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.389160 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ccfcacb5b294a628926d34cd31b137e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.389375 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 9ccfcacb5b294a628926d34cd31b137e (DEFAULT_TABLE table=default.Table_93954115dc5d4d2283faa424e83608b0 [id=cb30302a7db04098ae9a5a92ce055d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.397153 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ccfcacb5b294a628926d34cd31b137e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.382102 14714 ts_tablet_manager.cc:1434] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.064s user 0.007s sys 0.000s
I20251211 08:57:35.404914 14714 tablet_bootstrap.cc:492] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:35.395540 15030 raft_consensus.cc:359] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.408164 15030 raft_consensus.cc:385] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:35.409941 14714 tablet_bootstrap.cc:654] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:35.425405 14718 ts_tablet_manager.cc:1929] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:35.425789 14718 log.cc:1199] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/15f7bcd680064b9f8fe51aa05acce6fb
I20251211 08:57:35.426396 14718 ts_tablet_manager.cc:1950] T 15f7bcd680064b9f8fe51aa05acce6fb P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:35.442956 14006 master_service.cc:954] ReplaceTablet: received request to replace tablet c53035e83710403bac3a2b401c8c97ad from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:35.445082 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet db302282bc3d4d8cb56d1cbb2f73c4fe
I20251211 08:57:35.446241 14013 catalog_manager.cc:6724] ReplaceTablet: tablet db302282bc3d4d8cb56d1cbb2f73c4fe deleted and replaced by tablet f785a3cc2e8145eebae7e9c2885b0b1e
I20251211 08:57:35.454988 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet db302282bc3d4d8cb56d1cbb2f73c4fe with delete_type TABLET_DATA_DELETED (replaced by tablet f785a3cc2e8145eebae7e9c2885b0b1e) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:35.462976 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 15f7bcd680064b9f8fe51aa05acce6fb (table default.Table_d1d406b278304b5ca6e6cc3b63e7fd36 [id=4208ab42111742f38028caeb7913a1c7]) successfully deleted
I20251211 08:57:35.463691 14718 tablet_replica.cc:333] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:35.464355 14718 raft_consensus.cc:2243] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:35.464788 14718 raft_consensus.cc:2272] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:35.471496 14718 ts_tablet_manager.cc:1916] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:35.476500 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4960eb1f338e47b3923b57ce8be73703"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:35.478013 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4960eb1f338e47b3923b57ce8be73703 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:35.480723 13999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20251211 08:57:35.391283 14266 tablet_service.cc:1505] Processing CreateTablet for tablet f909d82e0136470a810e1063d6769868 (DEFAULT_TABLE table=default.Table_c15367b621c144348ec8511d2947fec6 [id=ea01e214ed314a26a0de20d34e37863b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.485839 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f909d82e0136470a810e1063d6769868. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.413282 14269 tablet_service.cc:1552] Processing DeleteTablet for tablet 25805a95349d49f58280a30e6876929d with delete_type TABLET_DATA_DELETED (replaced by tablet 41d6ef975a47489385791065bb8f9f46) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:35.450950 14279 tablet_service.cc:1552] Processing DeleteTablet for tablet db302282bc3d4d8cb56d1cbb2f73c4fe with delete_type TABLET_DATA_DELETED (replaced by tablet f785a3cc2e8145eebae7e9c2885b0b1e) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:35.479408 14904 ts_tablet_manager.cc:1929] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:35.491706 14904 log.cc:1199] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/60b23163906447c3877db631968f5623
I20251211 08:57:35.492306 14904 ts_tablet_manager.cc:1950] T 60b23163906447c3877db631968f5623 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:35.500965 14429 tablet_service.cc:1505] Processing CreateTablet for tablet f909d82e0136470a810e1063d6769868 (DEFAULT_TABLE table=default.Table_c15367b621c144348ec8511d2947fec6 [id=ea01e214ed314a26a0de20d34e37863b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.502203 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f909d82e0136470a810e1063d6769868. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.502393 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 4eec7694087145d8a49dd93904f95980 (DEFAULT_TABLE table=default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.503316 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eec7694087145d8a49dd93904f95980. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.504431 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 4eec7694087145d8a49dd93904f95980 (DEFAULT_TABLE table=default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.505543 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eec7694087145d8a49dd93904f95980. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.412549 15152 raft_consensus.cc:493] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:35.448961 14581 tablet_service.cc:1552] Processing DeleteTablet for tablet db302282bc3d4d8cb56d1cbb2f73c4fe with delete_type TABLET_DATA_DELETED (replaced by tablet f785a3cc2e8145eebae7e9c2885b0b1e) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.454638 15163 consensus_queue.cc:1048] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:35.507439 15152 raft_consensus.cc:515] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.510028 15152 leader_election.cc:290] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:35.511113 15030 raft_consensus.cc:740] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:35.511818 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf537c461c04a81b66f268ee98dc64e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:35.511806 15030 consensus_queue.cc:260] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.512625 14722 tablet_bootstrap.cc:492] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:35.513773 14581 tablet_service.cc:1505] Processing CreateTablet for tablet 4eec7694087145d8a49dd93904f95980 (DEFAULT_TABLE table=default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.514689 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eec7694087145d8a49dd93904f95980. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:35.519311 14544 leader_election.cc:343] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:35.520478 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf537c461c04a81b66f268ee98dc64e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:35.521767 14542 leader_election.cc:343] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:35.522128 14542 leader_election.cc:304] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:35.523707 14714 tablet_bootstrap.cc:492] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:35.524214 14714 ts_tablet_manager.cc:1403] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.120s user 0.014s sys 0.003s
I20251211 08:57:35.530197 15152 raft_consensus.cc:2749] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:35.532713 15030 ts_tablet_manager.cc:1434] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.145s user 0.006s sys 0.000s
I20251211 08:57:35.540922 14585 tablet_service.cc:1505] Processing CreateTablet for tablet f909d82e0136470a810e1063d6769868 (DEFAULT_TABLE table=default.Table_c15367b621c144348ec8511d2947fec6 [id=ea01e214ed314a26a0de20d34e37863b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:35.549511 14010 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb consensus state has no leader
I20251211 08:57:35.550470 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7919ce70cfc5400bb4f2515164c6b7a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:35.552155 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7919ce70cfc5400bb4f2515164c6b7a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:35.552634 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 6de43c843ff845a0bd4eb98b6054dd87 (table default.Table_3c7e457251e74eb7b8415bbcb6b6d7db [id=e5ffdeadfa904d85b68d0925da7e7a4c]). Expected version 1 got 0
I20251211 08:57:35.553293 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 40a1f12553934446bfe44f4ed41abf9a (table default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127]). Expected version 1 got 0
I20251211 08:57:35.554204 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 5cf1210733754f43885a7c0a350aa206 (table default.Table_2d7127602bd94e2dbc13d3cfae809a87 [id=e9973d6c9b1341d1b96f04367e44e212]). Expected version 2 got 0
I20251211 08:57:35.554737 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 3303dc521d694be6b386a35da1172ede (table default.Table_71b84842b7c84927ad89c5ad951be1a1 [id=ebfd6e4105a8410bb79a6db704ec0ba5]). Expected version 2 got 0
I20251211 08:57:35.555330 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet eb5a6e533e1f4da7b03a800d029f712d (table default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85]). Expected version 1 got 0
W20251211 08:57:35.566274 13999 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 -> Peer 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015): Couldn't send request to peer 73702d54989f4959acfc4a6fdcede6ca. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20251211 08:57:35.572659 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 5cf1210733754f43885a7c0a350aa206 (table default.Table_2d7127602bd94e2dbc13d3cfae809a87 [id=e9973d6c9b1341d1b96f04367e44e212]). Expected version 2 got 0
I20251211 08:57:35.573326 14013 catalog_manager.cc:5654] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:35.574579 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet eb5a6e533e1f4da7b03a800d029f712d (table default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85]). Expected version 1 got 0
I20251211 08:57:35.575336 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 3303dc521d694be6b386a35da1172ede (table default.Table_71b84842b7c84927ad89c5ad951be1a1 [id=ebfd6e4105a8410bb79a6db704ec0ba5]). Expected version 2 got 0
I20251211 08:57:35.576085 14013 catalog_manager.cc:5654] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:35.584740 15030 tablet_bootstrap.cc:492] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:35.602653 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_fcf104ee591d470d806156189c2cac52" } modify_external_catalogs: true
I20251211 08:57:35.603197 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_fcf104ee591d470d806156189c2cac52" } modify_external_catalogs: true
I20251211 08:57:35.604632 15083 raft_consensus.cc:493] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:35.605032 15083 raft_consensus.cc:515] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.608119 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3303dc521d694be6b386a35da1172ede" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:35.606702 15083 leader_election.cc:290] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.608668 14600 raft_consensus.cc:2468] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:35.608948 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3303dc521d694be6b386a35da1172ede" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:35.609512 14298 raft_consensus.cc:2468] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:35.609783 14382 leader_election.cc:304] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:35.611878 15083 raft_consensus.cc:2804] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:35.612207 15083 raft_consensus.cc:493] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:35.612519 15083 raft_consensus.cc:3060] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.624059 14425 tablet_service.cc:1505] Processing CreateTablet for tablet f5b462cf21e3420c8e323a4ac4f04abe (DEFAULT_TABLE table=default.Table_cf5e7d3231db4b53b9ba619336f5bb19 [id=6c43f429406443d7b5834905929dcbd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.625010 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b462cf21e3420c8e323a4ac4f04abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.634899 14714 raft_consensus.cc:359] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.635993 14714 raft_consensus.cc:385] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:35.636242 14714 raft_consensus.cc:740] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:35.636998 14714 consensus_queue.cc:260] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.642746 15182 raft_consensus.cc:493] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:35.643167 14905 ts_tablet_manager.cc:1929] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:35.643509 14905 log.cc:1199] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/a4e307e4d1994af0b99a88b3b06b73a4
I20251211 08:57:35.643187 15182 raft_consensus.cc:515] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.644073 14905 ts_tablet_manager.cc:1950] T a4e307e4d1994af0b99a88b3b06b73a4 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:35.644543 14722 tablet_bootstrap.cc:654] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:35.644809 15182 leader_election.cc:290] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.646966 14714 ts_tablet_manager.cc:1434] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.122s user 0.007s sys 0.001s
I20251211 08:57:35.647126 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f909d82e0136470a810e1063d6769868. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.647588 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c5315f708c04322919c714dc915a483" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:35.649384 14382 leader_election.cc:343] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:35.647753 14714 tablet_bootstrap.cc:492] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:35.658226 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 119) took 1948 ms. Trace:
I20251211 08:57:35.658517 14905 rpcz_store.cc:276] 1211 08:57:33.709456 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.709554 (+ 98us) service_pool.cc:225] Handling call
1211 08:57:35.307075 (+1597521us) ts_tablet_manager.cc:1151] Deleting tablet a4e307e4d1994af0b99a88b3b06b73a4
1211 08:57:35.307116 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.337535 (+ 30419us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:35.642964 (+305429us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:35.658146 (+ 15182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1596946}
I20251211 08:57:35.659507 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet a4e307e4d1994af0b99a88b3b06b73a4 (table default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]) successfully deleted
I20251211 08:57:35.660768 14266 tablet_service.cc:1505] Processing CreateTablet for tablet f5b462cf21e3420c8e323a4ac4f04abe (DEFAULT_TABLE table=default.Table_cf5e7d3231db4b53b9ba619336f5bb19 [id=6c43f429406443d7b5834905929dcbd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.660910 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c5315f708c04322919c714dc915a483" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:35.661916 14271 tablet_service.cc:1505] Processing CreateTablet for tablet e15483f539f24f1ba1b442663d66fd45 (DEFAULT_TABLE table=default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.662230 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b462cf21e3420c8e323a4ac4f04abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.662814 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15483f539f24f1ba1b442663d66fd45. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:35.663381 14385 leader_election.cc:343] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:35.663709 14385 leader_election.cc:304] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:35.666159 14580 tablet_service.cc:1505] Processing CreateTablet for tablet e15483f539f24f1ba1b442663d66fd45 (DEFAULT_TABLE table=default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.667260 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15483f539f24f1ba1b442663d66fd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.590713 15030 tablet_bootstrap.cc:654] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:35.676222 14298 raft_consensus.cc:3060] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.681914 14008 master_service.cc:954] ReplaceTablet: received request to replace tablet 479d7fe06a284234aef420e3842f0eab from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:35.690621 14905 tablet_replica.cc:333] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:35.689544 14269 tablet_service.cc:1505] Processing CreateTablet for tablet 90c8960c5bfa4a08905bd211654e4798 (DEFAULT_TABLE table=default.Table_7c905b0fe6de4e57bb2c49d7bf87206a [id=9f3e442a656a4a8080fbf847cb136b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.691520 14905 raft_consensus.cc:2243] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:35.691805 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c8960c5bfa4a08905bd211654e4798. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.691957 14905 raft_consensus.cc:2272] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:35.693274 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 127) took 1880 ms. Trace:
I20251211 08:57:35.693718 14905 ts_tablet_manager.cc:1916] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:35.693557 14904 rpcz_store.cc:276] 1211 08:57:33.812404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.826400 (+ 13996us) service_pool.cc:225] Handling call
1211 08:57:35.287639 (+1461239us) ts_tablet_manager.cc:1151] Deleting tablet 60b23163906447c3877db631968f5623
1211 08:57:35.287696 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.393894 (+106198us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:35.479208 (+ 85314us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:35.693218 (+214010us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1435125}
I20251211 08:57:35.694593 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 60b23163906447c3877db631968f5623 (table default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]) successfully deleted
I20251211 08:57:35.694716 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 126) took 1884 ms. Trace:
I20251211 08:57:35.694995 14904 rpcz_store.cc:276] 1211 08:57:33.810280 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.824531 (+ 14251us) service_pool.cc:225] Handling call
1211 08:57:35.694442 (+1869911us) ts_tablet_manager.cc:1151] Deleting tablet 426e26859b9c4966b8ec8ba3eab53a43
1211 08:57:35.694498 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.694673 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1841563}
I20251211 08:57:35.696018 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 131) took 1765 ms. Trace:
I20251211 08:57:35.696287 14904 rpcz_store.cc:276] 1211 08:57:33.930405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.936558 (+ 6153us) service_pool.cc:225] Handling call
1211 08:57:35.695722 (+1759164us) ts_tablet_manager.cc:1151] Deleting tablet caf537c461c04a81b66f268ee98dc64e
1211 08:57:35.695764 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.695979 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692003}
I20251211 08:57:35.697196 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 134) took 1206 ms. Trace:
I20251211 08:57:35.697454 14904 rpcz_store.cc:276] 1211 08:57:34.490414 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.495027 (+ 4613us) service_pool.cc:225] Handling call
1211 08:57:35.696929 (+1201902us) ts_tablet_manager.cc:1151] Deleting tablet 3c5315f708c04322919c714dc915a483
1211 08:57:35.696962 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.697155 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1199203}
I20251211 08:57:35.698295 14904 tablet_replica.cc:333] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b: stopping tablet replica
W20251211 08:57:35.699071 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 426e26859b9c4966b8ec8ba3eab53a43 because the tablet was not found. No further retry: Not found: Tablet not found: 426e26859b9c4966b8ec8ba3eab53a43
W20251211 08:57:35.699931 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet caf537c461c04a81b66f268ee98dc64e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caf537c461c04a81b66f268ee98dc64e already in progress: creating tablet
I20251211 08:57:35.712464 15030 tablet_bootstrap.cc:492] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:35.715457 15030 ts_tablet_manager.cc:1403] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.131s user 0.010s sys 0.000s
I20251211 08:57:35.717195 14718 ts_tablet_manager.cc:1929] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:35.717530 14718 log.cc:1199] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/25805a95349d49f58280a30e6876929d
I20251211 08:57:35.726562 15159 consensus_queue.cc:1048] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:35.731254 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 42720f5f5bf445f2b72e513ad4a6d95f
I20251211 08:57:35.731506 14584 tablet_service.cc:1505] Processing CreateTablet for tablet f5b462cf21e3420c8e323a4ac4f04abe (DEFAULT_TABLE table=default.Table_cf5e7d3231db4b53b9ba619336f5bb19 [id=6c43f429406443d7b5834905929dcbd4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:35.737960 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4f826d079774152a3b66bfb81f990a8 consensus state has no leader
I20251211 08:57:35.738499 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b462cf21e3420c8e323a4ac4f04abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.742358 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet c53035e83710403bac3a2b401c8c97ad
I20251211 08:57:35.743793 14006 catalog_manager.cc:6724] ReplaceTablet: tablet c53035e83710403bac3a2b401c8c97ad deleted and replaced by tablet dd14de369e084828afb8a924c46c649b
I20251211 08:57:35.747444 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 90c8960c5bfa4a08905bd211654e4798 (DEFAULT_TABLE table=default.Table_7c905b0fe6de4e57bb2c49d7bf87206a [id=9f3e442a656a4a8080fbf847cb136b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.748487 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c8960c5bfa4a08905bd211654e4798. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.749351 14722 tablet_bootstrap.cc:492] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:35.749716 14722 ts_tablet_manager.cc:1403] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.251s user 0.003s sys 0.007s
I20251211 08:57:35.751008 14577 tablet_service.cc:1552] Processing DeleteTablet for tablet c53035e83710403bac3a2b401c8c97ad with delete_type TABLET_DATA_DELETED (replaced by tablet dd14de369e084828afb8a924c46c649b) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.751883 14722 raft_consensus.cc:359] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.752468 14722 raft_consensus.cc:385] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:35.752703 14722 raft_consensus.cc:740] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:35.753217 14722 consensus_queue.cc:260] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.752935 14577 tablet_service.cc:1505] Processing CreateTablet for tablet 4e813a4c73334c839bfe6af2119e259c (DEFAULT_TABLE table=default.Table_358b3598ddbe4cc2831ba06d3600ac09 [id=ffaa7c2a60924b0cbdc1254bfad98189]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.757858 15163 consensus_queue.cc:1048] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:35.759712 14429 tablet_service.cc:1505] Processing CreateTablet for tablet e15483f539f24f1ba1b442663d66fd45 (DEFAULT_TABLE table=default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.760797 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15483f539f24f1ba1b442663d66fd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.762758 15030 raft_consensus.cc:359] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.763242 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e813a4c73334c839bfe6af2119e259c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.763444 15030 raft_consensus.cc:385] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:35.763695 15030 raft_consensus.cc:740] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:35.764288 15030 consensus_queue.cc:260] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.718139 14718 ts_tablet_manager.cc:1950] T 25805a95349d49f58280a30e6876929d P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:35.768528 14718 tablet_replica.cc:333] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:35.769258 14718 raft_consensus.cc:2243] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:35.769709 14718 raft_consensus.cc:2272] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:35.774881 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a9bd9e30e1ee4c92bbdd8e1748dd0ddd" } new_table_name: "default.Table_a4bc9b5633444f8587bf740793fd55a2" modify_external_catalogs: true
I20251211 08:57:35.775981 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 90c8960c5bfa4a08905bd211654e4798 (DEFAULT_TABLE table=default.Table_7c905b0fe6de4e57bb2c49d7bf87206a [id=9f3e442a656a4a8080fbf847cb136b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.776960 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c8960c5bfa4a08905bd211654e4798. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.780071 14279 tablet_service.cc:1552] Processing DeleteTablet for tablet c53035e83710403bac3a2b401c8c97ad with delete_type TABLET_DATA_DELETED (replaced by tablet dd14de369e084828afb8a924c46c649b) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:35.754617 14722 ts_tablet_manager.cc:1434] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251211 08:57:35.793503 14722 tablet_bootstrap.cc:492] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:35.793186 14011 catalog_manager.cc:5654] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:35.794267 14011 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 6de43c843ff845a0bd4eb98b6054dd87 (table default.Table_3c7e457251e74eb7b8415bbcb6b6d7db [id=e5ffdeadfa904d85b68d0925da7e7a4c]). Expected version 1 got 0
I20251211 08:57:35.797358 14011 catalog_manager.cc:5654] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:35.799490 14011 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 3303dc521d694be6b386a35da1172ede (table default.Table_71b84842b7c84927ad89c5ad951be1a1 [id=ebfd6e4105a8410bb79a6db704ec0ba5]). Expected version 2 got 0
I20251211 08:57:35.800405 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 42720f5f5bf445f2b72e513ad4a6d95f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:35.801115 14011 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet eb5a6e533e1f4da7b03a800d029f712d (table default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85]). Expected version 1 got 0
I20251211 08:57:35.801996 14579 tablet_service.cc:1505] Processing CreateTablet for tablet c0e0ee99989d41c6af82e330ab8e353c (DEFAULT_TABLE table=default.Table_7ca1c0fc20ca4dd7a11793c0ac968666 [id=67ab831326db412fa228aac2683950ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.803066 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e0ee99989d41c6af82e330ab8e353c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.803494 15030 ts_tablet_manager.cc:1434] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.088s user 0.001s sys 0.004s
I20251211 08:57:35.804353 15030 tablet_bootstrap.cc:492] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:35.804507 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet c53035e83710403bac3a2b401c8c97ad with delete_type TABLET_DATA_DELETED (replaced by tablet dd14de369e084828afb8a924c46c649b) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:35.804095 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 4e813a4c73334c839bfe6af2119e259c (DEFAULT_TABLE table=default.Table_358b3598ddbe4cc2831ba06d3600ac09 [id=ffaa7c2a60924b0cbdc1254bfad98189]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.805575 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e813a4c73334c839bfe6af2119e259c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.811060 14580 tablet_service.cc:1505] Processing CreateTablet for tablet f938359354b545cfbac560b7d7888b9f (DEFAULT_TABLE table=default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.772122 15182 raft_consensus.cc:2749] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:35.761461 15192 raft_consensus.cc:493] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:35.640148 15083 raft_consensus.cc:515] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.815413 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 4e813a4c73334c839bfe6af2119e259c (DEFAULT_TABLE table=default.Table_358b3598ddbe4cc2831ba06d3600ac09 [id=ffaa7c2a60924b0cbdc1254bfad98189]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.816315 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e813a4c73334c839bfe6af2119e259c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.816300 14425 tablet_service.cc:1505] Processing CreateTablet for tablet f938359354b545cfbac560b7d7888b9f (DEFAULT_TABLE table=default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.817227 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f938359354b545cfbac560b7d7888b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.819532 14904 raft_consensus.cc:2243] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:35.820137 14904 raft_consensus.cc:2272] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:35.821935 14904 ts_tablet_manager.cc:1916] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:35.821942 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f938359354b545cfbac560b7d7888b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.823594 14273 tablet_service.cc:1505] Processing CreateTablet for tablet f938359354b545cfbac560b7d7888b9f (DEFAULT_TABLE table=default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.832412 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 25805a95349d49f58280a30e6876929d (table default.Table_fa2830322a8a4f7b96cd19543b967b05 [id=52ae4637827d424eb4d37362bf21cea2]) successfully deleted
I20251211 08:57:35.840170 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 (DEFAULT_TABLE table=default.Table_d8ac34f0cab345dd9b113ef1447ea361 [id=16e3304bc6774f7dab11c7ebd6eaa771]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.842130 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 (DEFAULT_TABLE table=default.Table_d8ac34f0cab345dd9b113ef1447ea361 [id=16e3304bc6774f7dab11c7ebd6eaa771]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.843523 14576 tablet_service.cc:1505] Processing CreateTablet for tablet 8c39cc816f4c438391bb4826ea1a508f (DEFAULT_TABLE table=default.Table_fc4825b70b434d46a0295dc99cbbb4c2 [id=437ef7fe2007480eab5c5fdc96d1b1fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.844152 14581 tablet_service.cc:1552] Processing DeleteTablet for tablet 42720f5f5bf445f2b72e513ad4a6d95f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.844858 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.845369 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_25fc063a532541f5a7d6c73d8f2f0e33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:35.846733 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25fc063a532541f5a7d6c73d8f2f0e33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:35.849851 14422 tablet_service.cc:1505] Processing CreateTablet for tablet 8c39cc816f4c438391bb4826ea1a508f (DEFAULT_TABLE table=default.Table_fc4825b70b434d46a0295dc99cbbb4c2 [id=437ef7fe2007480eab5c5fdc96d1b1fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.847384 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_cb9f538930f64a52b0f18805535a8fc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:35.851846 14420 tablet_service.cc:1505] Processing CreateTablet for tablet c0e0ee99989d41c6af82e330ab8e353c (DEFAULT_TABLE table=default.Table_7ca1c0fc20ca4dd7a11793c0ac968666 [id=67ab831326db412fa228aac2683950ee]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:35.852396 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb9f538930f64a52b0f18805535a8fc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:35.822811 15192 raft_consensus.cc:515] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.854199 15192 leader_election.cc:290] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:35.854979 14382 consensus_peers.cc:597] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60b23163906447c3877db631968f5623. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:35.851107 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.857055 15083 leader_election.cc:290] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.857652 14272 tablet_service.cc:1505] Processing CreateTablet for tablet c0e0ee99989d41c6af82e330ab8e353c (DEFAULT_TABLE table=default.Table_7ca1c0fc20ca4dd7a11793c0ac968666 [id=67ab831326db412fa228aac2683950ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.858489 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3303dc521d694be6b386a35da1172ede" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:35.859118 14606 raft_consensus.cc:3060] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.863006 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c39cc816f4c438391bb4826ea1a508f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.867128 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e0ee99989d41c6af82e330ab8e353c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.871083 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c39cc816f4c438391bb4826ea1a508f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.875733 15083 raft_consensus.cc:493] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:35.876152 15083 raft_consensus.cc:515] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.878317 15083 leader_election.cc:290] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:35.879665 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1976d584cc0a450ab809ead258a251e2" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:35.880224 14604 raft_consensus.cc:2468] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:35.883134 14382 leader_election.cc:304] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
W20251211 08:57:35.887851 14382 leader_election.cc:343] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: a4e307e4d1994af0b99a88b3b06b73a4
I20251211 08:57:35.891714 14606 raft_consensus.cc:2468] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:35.892727 14382 leader_election.cc:304] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:35.893486 15192 raft_consensus.cc:2804] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:35.893827 15192 raft_consensus.cc:493] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:35.894107 15192 raft_consensus.cc:3060] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:35.898927 15083 raft_consensus.cc:2804] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:35.899317 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 479d7fe06a284234aef420e3842f0eab
I20251211 08:57:35.899410 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 (DEFAULT_TABLE table=default.Table_d8ac34f0cab345dd9b113ef1447ea361 [id=16e3304bc6774f7dab11c7ebd6eaa771]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.899691 15083 raft_consensus.cc:697] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:35.900620 14008 catalog_manager.cc:6724] ReplaceTablet: tablet 479d7fe06a284234aef420e3842f0eab deleted and replaced by tablet f5c5b1aebde94dea81cb37a6a6998070
I20251211 08:57:35.900450 15083 consensus_queue.cc:237] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.903162 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 174fefd70bcc41418a77a5716f97cd5f
I20251211 08:57:35.907310 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet 174fefd70bcc41418a77a5716f97cd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:35.911053 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 174fefd70bcc41418a77a5716f97cd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.918431 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet 479d7fe06a284234aef420e3842f0eab with delete_type TABLET_DATA_DELETED (replaced by tablet f5c5b1aebde94dea81cb37a6a6998070) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.918993 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f938359354b545cfbac560b7d7888b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.919739 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.923192 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e0ee99989d41c6af82e330ab8e353c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.937578 14009 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_f0670566e5df45448e1786b9c82156d4 [id=491e5bc8d1524a7aae54a99a4d39c7f2] alter complete (version 1)
I20251211 08:57:35.939271 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_face70f737874a76a60d1f4ea105380c" } modify_external_catalogs: true
I20251211 08:57:35.940430 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_face70f737874a76a60d1f4ea105380c" } modify_external_catalogs: true
I20251211 08:57:35.942873 14718 ts_tablet_manager.cc:1916] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:35.943370 14269 tablet_service.cc:1552] Processing DeleteTablet for tablet 42720f5f5bf445f2b72e513ad4a6d95f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:35.945207 14268 tablet_service.cc:1505] Processing CreateTablet for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (DEFAULT_TABLE table=default.Table_90208748f83c43e9b2bf122c5425257e [id=bf934f83df6142bf8788f1cd2f2142c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:35.946009 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b9b338a3f544ccb813247250b0f0eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.946709 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:35.946744 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e307e4d1994af0b99a88b3b06b73a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec7694087145d8a49dd93904f95980) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:35.950949 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ec0af779c90941db9c7bf85ccff7499f" } new_table_name: "default.Table_baaa95f73b8742388a79269d898a5a17" modify_external_catalogs: true
I20251211 08:57:35.955833 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 479d7fe06a284234aef420e3842f0eab with delete_type TABLET_DATA_DELETED (replaced by tablet f5c5b1aebde94dea81cb37a6a6998070) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:35.930238 15192 raft_consensus.cc:515] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:35.986253 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1976d584cc0a450ab809ead258a251e2" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:35.986909 14606 raft_consensus.cc:3060] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:35.992138 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98d14ba15dcd47f8b9e8035b3c7d477c has no consensus state
I20251211 08:57:35.993764 14606 raft_consensus.cc:2468] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:35.942689 14714 tablet_bootstrap.cc:654] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:35.996043 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_cfa8f9f640e54badacad03895e72ca57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:35.997663 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfa8f9f640e54badacad03895e72ca57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:36.000629 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f0e893169e89418ab044ae00cdb78176"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:36.002198 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0e893169e89418ab044ae00cdb78176 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:36.005471 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c5315f708c04322919c714dc915a483 with delete_type TABLET_DATA_DELETED (replaced by tablet 69aae3e1d7f34c70b0b48ce63089b482) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:36.006880 15192 leader_election.cc:290] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:36.008848 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 1976d584cc0a450ab809ead258a251e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:36.013947 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 5cf1210733754f43885a7c0a350aa206 (table default.Table_2d7127602bd94e2dbc13d3cfae809a87 [id=e9973d6c9b1341d1b96f04367e44e212]). Expected version 2 got 0
I20251211 08:57:36.034082 15193 raft_consensus.cc:493] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:36.034504 15193 raft_consensus.cc:515] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.036268 15193 leader_election.cc:290] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:36.039029 15159 consensus_queue.cc:1048] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:36.040436 15158 raft_consensus.cc:493] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:36.040863 15158 raft_consensus.cc:515] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.042654 15158 leader_election.cc:290] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:36.043800 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cf1210733754f43885a7c0a350aa206" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:36.046514 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 174fefd70bcc41418a77a5716f97cd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:36.047525 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:31 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:36.048885 14714 tablet_bootstrap.cc:492] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:36.049791 14714 ts_tablet_manager.cc:1403] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.402s user 0.010s sys 0.000s
I20251211 08:57:36.053121 14714 raft_consensus.cc:359] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.053936 14714 raft_consensus.cc:385] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:36.054226 14714 raft_consensus.cc:740] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:35.964202 14269 tablet_service.cc:1505] Processing CreateTablet for tablet 8c39cc816f4c438391bb4826ea1a508f (DEFAULT_TABLE table=default.Table_fc4825b70b434d46a0295dc99cbbb4c2 [id=437ef7fe2007480eab5c5fdc96d1b1fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:36.064772 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c39cc816f4c438391bb4826ea1a508f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:36.059340 15213 tablet.cc:1722] T 5cf1210733754f43885a7c0a350aa206 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:36.067721 14722 tablet_bootstrap.cc:700] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b: Time spent opening tablet: real 0.153s user 0.001s sys 0.000s
I20251211 08:57:36.068331 14722 tablet_bootstrap.cc:654] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:36.068257 14714 consensus_queue.cc:260] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.072089 15148 raft_consensus.cc:493] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:36.072537 15148 raft_consensus.cc:515] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.073347 14279 tablet_service.cc:1552] Processing DeleteTablet for tablet 479d7fe06a284234aef420e3842f0eab with delete_type TABLET_DATA_DELETED (replaced by tablet f5c5b1aebde94dea81cb37a6a6998070) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:36.074425 15148 leader_election.cc:290] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:36.044251 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81cfb3e0baef45ddab4a77a4e0790085" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:36.075160 14604 raft_consensus.cc:2468] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:36.044833 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cf1210733754f43885a7c0a350aa206" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:36.063796 15214 tablet.cc:1722] T 5cf1210733754f43885a7c0a350aa206 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:36.076162 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6de43c843ff845a0bd4eb98b6054dd87" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:36.067484 15030 tablet_bootstrap.cc:654] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:36.077553 14230 leader_election.cc:304] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:36.079061 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6de43c843ff845a0bd4eb98b6054dd87" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:36.079847 14600 raft_consensus.cc:2393] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:36.076192 14714 ts_tablet_manager.cc:1434] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.026s user 0.003s sys 0.004s
I20251211 08:57:36.081053 14714 tablet_bootstrap.cc:492] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:36.081056 14230 leader_election.cc:304] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:36.082358 15148 raft_consensus.cc:2749] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:36.082919 15148 raft_consensus.cc:3060] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:36.085798 14382 leader_election.cc:304] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:36.087172 15122 raft_consensus.cc:2804] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:36.087467 15122 raft_consensus.cc:493] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:36.087779 15122 raft_consensus.cc:3060] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:36.089397 15148 raft_consensus.cc:2749] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:36.090996 14382 leader_election.cc:304] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:36.092543 15193 raft_consensus.cc:2804] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:36.092985 15193 raft_consensus.cc:697] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:36.093686 15193 consensus_queue.cc:237] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.094239 15122 raft_consensus.cc:515] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.096299 15122 leader_election.cc:290] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:36.099390 14013 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b5a4b6b1b28406c82e2acd9356b60d4 consensus state has no leader
I20251211 08:57:36.110076 14718 ts_tablet_manager.cc:1929] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:36.110456 14718 log.cc:1199] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/2cd6e4352cfc41e497dc58d158e24ff2
I20251211 08:57:36.113814 14714 tablet_bootstrap.cc:654] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:36.114220 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81cfb3e0baef45ddab4a77a4e0790085" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:36.114835 14606 raft_consensus.cc:3060] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:36.116823 15152 raft_consensus.cc:493] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:36.117307 15152 raft_consensus.cc:515] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.119151 15152 leader_election.cc:290] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:36.120622 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8d753358f20476ba23d7d7b72e26e74" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:36.121953 14544 leader_election.cc:343] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:36.125797 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_baaa95f73b8742388a79269d898a5a17" } modify_external_catalogs: true
I20251211 08:57:36.126262 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_baaa95f73b8742388a79269d898a5a17" } modify_external_catalogs: true
I20251211 08:57:36.130636 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (DEFAULT_TABLE table=default.Table_90208748f83c43e9b2bf122c5425257e [id=bf934f83df6142bf8788f1cd2f2142c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:36.133260 14606 raft_consensus.cc:2468] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:36.138978 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3303dc521d694be6b386a35da1172ede" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:36.139483 14298 raft_consensus.cc:3060] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:36.142442 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e307e4d1994af0b99a88b3b06b73a4" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:36.143338 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1976d584cc0a450ab809ead258a251e2" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:36.146288 14298 raft_consensus.cc:2468] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:36.145767 14013 catalog_manager.cc:5654] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:36.152702 14013 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 3303dc521d694be6b386a35da1172ede (table default.Table_71b84842b7c84927ad89c5ad951be1a1 [id=ebfd6e4105a8410bb79a6db704ec0ba5]). Expected version 2 got 0
I20251211 08:57:36.153489 14013 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 5cf1210733754f43885a7c0a350aa206 (table default.Table_2d7127602bd94e2dbc13d3cfae809a87 [id=e9973d6c9b1341d1b96f04367e44e212]). Expected version 2 got 0
I20251211 08:57:36.158200 14904 ts_tablet_manager.cc:1929] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:36.158681 14904 log.cc:1199] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/a0f9c321e673433498a9059dd2e2efe6
I20251211 08:57:36.172554 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:36.177762 14722 tablet_bootstrap.cc:492] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:36.178165 14722 ts_tablet_manager.cc:1403] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.385s user 0.007s sys 0.004s
I20251211 08:57:36.180305 14722 raft_consensus.cc:359] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.180907 14722 raft_consensus.cc:385] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:36.181169 14722 raft_consensus.cc:740] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:36.184649 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1976d584cc0a450ab809ead258a251e2" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:36.186218 14297 raft_consensus.cc:3060] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:36.204726 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81cfb3e0baef45ddab4a77a4e0790085" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:36.206015 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81cfb3e0baef45ddab4a77a4e0790085" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:36.119779 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8d753358f20476ba23d7d7b72e26e74" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:36.122929 14718 ts_tablet_manager.cc:1950] T 2cd6e4352cfc41e497dc58d158e24ff2 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
W20251211 08:57:36.209625 14542 leader_election.cc:343] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:36.210155 14542 leader_election.cc:304] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:36.211372 14718 tablet_replica.cc:333] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:36.211974 15152 raft_consensus.cc:2749] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:36.212163 14718 raft_consensus.cc:2243] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:36.212821 14718 pending_rounds.cc:70] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:36.213054 14718 pending_rounds.cc:77] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256390123106304 op_type: NO_OP noop_request { }
I20251211 08:57:36.213353 14718 raft_consensus.cc:2889] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:36.213635 14718 raft_consensus.cc:2272] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:36.134298 14382 leader_election.cc:304] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:36.148016 14429 tablet_service.cc:1505] Processing CreateTablet for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (DEFAULT_TABLE table=default.Table_90208748f83c43e9b2bf122c5425257e [id=bf934f83df6142bf8788f1cd2f2142c0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:36.215166 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 80983.3us
I20251211 08:57:36.215526 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:36.215780 15122 raft_consensus.cc:2804] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:36.216157 15122 raft_consensus.cc:697] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:36.216780 15122 consensus_queue.cc:237] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.219600 14904 ts_tablet_manager.cc:1950] T a0f9c321e673433498a9059dd2e2efe6 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:57:36.207463 14385 leader_election.cc:343] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:36.221699 14385 leader_election.cc:304] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:36.222170 14905 ts_tablet_manager.cc:1929] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:36.222477 14905 log.cc:1199] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/c181adf9db574ab39432b2f9b14fe4d7
I20251211 08:57:36.222699 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 2cd6e4352cfc41e497dc58d158e24ff2 (table default.Table_9c85250afba741a79335663257915c82 [id=48f11a87431349f8a3caeb458f06869d]) successfully deleted
W20251211 08:57:36.223099 14385 leader_election.cc:343] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:36.223801 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 2016208a33dc4a5cadd7ced8e85ccc65
I20251211 08:57:36.224057 15182 raft_consensus.cc:2749] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:36.225754 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 2016208a33dc4a5cadd7ced8e85ccc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:36.226454 14576 tablet_service.cc:1552] Processing DeleteTablet for tablet 2016208a33dc4a5cadd7ced8e85ccc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:36.232249 13999 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 (table default.Table_baaa95f73b8742388a79269d898a5a17 [id=99243463ed7545ff80f928e6664a3748], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9b5a4b6b1b28406c82e2acd9356b60d4 consensus state has no leader
I20251211 08:57:36.232959 14718 ts_tablet_manager.cc:1916] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:36.235702 14905 ts_tablet_manager.cc:1950] T c181adf9db574ab39432b2f9b14fe4d7 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:36.238049 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 120) took 2510 ms. Trace:
W20251211 08:57:36.241633 14385 leader_election.cc:343] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:36.242424 14385 leader_election.cc:343] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:36.243167 14385 leader_election.cc:343] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:36.248943 15159 consensus_queue.cc:1048] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:36.222044 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 135) took 1726 ms. Trace:
I20251211 08:57:36.252349 14904 rpcz_store.cc:276] 1211 08:57:34.495988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.536392 (+ 40404us) service_pool.cc:225] Handling call
1211 08:57:35.698175 (+1161783us) ts_tablet_manager.cc:1151] Deleting tablet a0f9c321e673433498a9059dd2e2efe6
1211 08:57:35.698208 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.965460 (+267252us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:36.157995 (+192535us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:36.221909 (+ 63914us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161161}
I20251211 08:57:36.253505 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 136) took 1716 ms. Trace:
I20251211 08:57:36.253767 14904 rpcz_store.cc:276] 1211 08:57:34.537365 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.537448 (+ 83us) service_pool.cc:225] Handling call
1211 08:57:36.253193 (+1715745us) ts_tablet_manager.cc:1151] Deleting tablet 5daba165262e41af8d3481d3b4929092
1211 08:57:36.253231 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.253456 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1715235}
I20251211 08:57:36.262619 14279 tablet_service.cc:1552] Processing DeleteTablet for tablet 2016208a33dc4a5cadd7ced8e85ccc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:36.264737 14722 consensus_queue.cc:260] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.268355 14722 ts_tablet_manager.cc:1434] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.090s user 0.006s sys 0.001s
I20251211 08:57:36.283365 14718 ts_tablet_manager.cc:1929] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:36.283524 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet a0f9c321e673433498a9059dd2e2efe6 (table default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5]) successfully deleted
I20251211 08:57:36.283773 14718 log.cc:1199] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/60b23163906447c3877db631968f5623
W20251211 08:57:36.284191 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 5daba165262e41af8d3481d3b4929092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5daba165262e41af8d3481d3b4929092 already in progress: creating tablet
I20251211 08:57:36.284453 14718 ts_tablet_manager.cc:1950] T 60b23163906447c3877db631968f5623 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:36.285126 14008 master_service.cc:954] ReplaceTablet: received request to replace tablet 8ad600554c0946e0bcb0da6a42be2ffd from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:36.286805 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 60b23163906447c3877db631968f5623 (table default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]) successfully deleted
W20251211 08:57:36.288210 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet db302282bc3d4d8cb56d1cbb2f73c4fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db302282bc3d4d8cb56d1cbb2f73c4fe already in progress: creating tablet
W20251211 08:57:36.288763 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 42720f5f5bf445f2b72e513ad4a6d95f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42720f5f5bf445f2b72e513ad4a6d95f already in progress: creating tablet
I20251211 08:57:36.290858 14718 tablet_replica.cc:333] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:36.291555 14718 raft_consensus.cc:2243] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:36.292347 14718 raft_consensus.cc:2272] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:36.292392 15083 consensus_queue.cc:1048] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:36.295285 15030 tablet_bootstrap.cc:492] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:36.295691 15030 ts_tablet_manager.cc:1403] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.492s user 0.012s sys 0.003s
I20251211 08:57:36.238350 14905 rpcz_store.cc:276] 1211 08:57:33.728079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.728836 (+ 757us) service_pool.cc:225] Handling call
1211 08:57:35.690446 (+1961610us) ts_tablet_manager.cc:1151] Deleting tablet c181adf9db574ab39432b2f9b14fe4d7
1211 08:57:35.690498 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:35.828335 (+137837us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:36.221969 (+393634us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:36.237999 (+ 16030us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1961068}
I20251211 08:57:36.307672 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_305ab52f2cd144b4b524e8deb6e1264f" } new_table_name: "default.Table_dda8864ad51542599828719c4cb52e5b" modify_external_catalogs: true
I20251211 08:57:36.308439 14714 tablet_bootstrap.cc:492] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:36.308838 14714 ts_tablet_manager.cc:1403] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.228s user 0.008s sys 0.001s
W20251211 08:57:36.328147 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet c53035e83710403bac3a2b401c8c97ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c53035e83710403bac3a2b401c8c97ad already in progress: creating tablet
W20251211 08:57:36.329667 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 174fefd70bcc41418a77a5716f97cd5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 174fefd70bcc41418a77a5716f97cd5f already in progress: creating tablet
I20251211 08:57:36.333845 14714 raft_consensus.cc:359] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.334460 14714 raft_consensus.cc:385] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:36.334703 14714 raft_consensus.cc:740] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:36.336056 14718 ts_tablet_manager.cc:1916] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:36.335259 14714 consensus_queue.cc:260] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.347870 15193 consensus_queue.cc:1048] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:36.350024 14714 ts_tablet_manager.cc:1434] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20251211 08:57:36.350924 14714 tablet_bootstrap.cc:492] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:36.355732 15193 consensus_queue.cc:1048] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:36.358378 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_58973fbd22514d66804dc101b76a7451"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:36.364383 14010 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 9b5a4b6b1b28406c82e2acd9356b60d4
W20251211 08:57:36.364432 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58973fbd22514d66804dc101b76a7451 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:36.370046 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:36.370537 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:36.370940 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 6de43c843ff845a0bd4eb98b6054dd87 (table default.Table_3c7e457251e74eb7b8415bbcb6b6d7db [id=e5ffdeadfa904d85b68d0925da7e7a4c]). Expected version 1 got 0
I20251211 08:57:36.372527 14714 tablet_bootstrap.cc:654] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:36.377228 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:31 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:36.377921 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e307e4d1994af0b99a88b3b06b73a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec7694087145d8a49dd93904f95980) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:36.377751 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7d2f7854e92d464cbebec8b6406ab053"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:36.371549 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 5cf1210733754f43885a7c0a350aa206 (table default.Table_2d7127602bd94e2dbc13d3cfae809a87 [id=e9973d6c9b1341d1b96f04367e44e212]). Expected version 2 got 0
W20251211 08:57:36.379161 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d2f7854e92d464cbebec8b6406ab053 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:36.383193 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b225e6a7b22c43e1ae401c96cf7eaad8" } new_table_name: "default.Table_da9fb81b38a14423b6191db9f7a7c3df" modify_external_catalogs: true
I20251211 08:57:36.383689 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet ad903727f79a4180adf047c92d981c7e from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:36.391202 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet c181adf9db574ab39432b2f9b14fe4d7 (table default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]) successfully deleted
I20251211 08:57:36.392231 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 121) took 2653 ms. Trace:
I20251211 08:57:36.392582 14905 rpcz_store.cc:276] 1211 08:57:33.738346 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.739778 (+ 1432us) service_pool.cc:225] Handling call
1211 08:57:36.391848 (+2652070us) ts_tablet_manager.cc:1151] Deleting tablet e961c5a3b5f640cbaea78b6185048d04
1211 08:57:36.391961 (+ 113us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.392183 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2651561}
W20251211 08:57:36.393908 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet e961c5a3b5f640cbaea78b6185048d04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e961c5a3b5f640cbaea78b6185048d04 already in progress: creating tablet
I20251211 08:57:36.394690 14905 tablet_replica.cc:333] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:36.395521 14905 raft_consensus.cc:2243] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:36.396096 14905 raft_consensus.cc:2272] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:36.398151 14905 ts_tablet_manager.cc:1916] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:36.406390 14718 ts_tablet_manager.cc:1929] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:36.413394 14718 log.cc:1199] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/479d7fe06a284234aef420e3842f0eab
I20251211 08:57:36.414047 14718 ts_tablet_manager.cc:1950] T 479d7fe06a284234aef420e3842f0eab P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:36.289654 15159 consensus_queue.cc:1048] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:36.312248 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 137) took 1546 ms. Trace:
I20251211 08:57:36.415004 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 479d7fe06a284234aef420e3842f0eab (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]) successfully deleted
I20251211 08:57:36.415015 14904 rpcz_store.cc:276] 1211 08:57:34.765407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.765525 (+ 118us) service_pool.cc:225] Handling call
1211 08:57:36.254442 (+1488917us) ts_tablet_manager.cc:1151] Deleting tablet 1976d584cc0a450ab809ead258a251e2
1211 08:57:36.289935 (+ 35493us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.312179 (+ 22244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1488295}
I20251211 08:57:36.355840 15226 tablet.cc:1722] T 5cf1210733754f43885a7c0a350aa206 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:36.415818 15159 consensus_queue.cc:1048] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:36.383898 14268 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:36.417826 14718 tablet_replica.cc:333] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:36.419384 14718 raft_consensus.cc:2243] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:36.420367 14718 raft_consensus.cc:2272] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:36.425024 14718 ts_tablet_manager.cc:1916] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:36.442317 14718 ts_tablet_manager.cc:1929] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:36.442981 14718 log.cc:1199] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/3c5315f708c04322919c714dc915a483
W20251211 08:57:36.446782 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 1976d584cc0a450ab809ead258a251e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1976d584cc0a450ab809ead258a251e2 already in progress: creating tablet
I20251211 08:57:36.448675 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_874fb7be95234d289bc2b30e873dabe9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:36.450192 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_874fb7be95234d289bc2b30e873dabe9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:36.451728 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_02e79cde30df4a2d8e81657161c2a56e" } modify_external_catalogs: true
I20251211 08:57:36.450486 15193 consensus_queue.cc:1048] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:36.452198 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_02e79cde30df4a2d8e81657161c2a56e" } modify_external_catalogs: true
I20251211 08:57:36.453720 14905 ts_tablet_manager.cc:1929] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:36.454017 14905 log.cc:1199] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/caf537c461c04a81b66f268ee98dc64e
I20251211 08:57:36.454587 14905 ts_tablet_manager.cc:1950] T caf537c461c04a81b66f268ee98dc64e P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:36.455555 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_c65937239ed54bb892938599e5f28043"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:36.456875 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c65937239ed54bb892938599e5f28043 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:36.461118 14718 ts_tablet_manager.cc:1950] T 3c5315f708c04322919c714dc915a483 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:36.464816 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 138) took 1648 ms. Trace:
I20251211 08:57:36.465091 14904 rpcz_store.cc:276] 1211 08:57:34.816354 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.816475 (+ 121us) service_pool.cc:225] Handling call
1211 08:57:36.416162 (+1599687us) ts_tablet_manager.cc:1151] Deleting tablet a6570281e1064c79b6e2297723f28658
1211 08:57:36.464532 (+ 48370us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.464769 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599156}
I20251211 08:57:36.465582 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 3c5315f708c04322919c714dc915a483 (table default.Table_d12516554d1547419ff004d016cb9a3a [id=1078a3f3dca34fbe8d48d184f434c7b6]) successfully deleted
I20251211 08:57:36.466370 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 142) took 1492 ms. Trace:
I20251211 08:57:36.466674 14718 tablet_replica.cc:333] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:36.466632 14904 rpcz_store.cc:276] 1211 08:57:34.974305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.974518 (+ 213us) service_pool.cc:225] Handling call
1211 08:57:36.466082 (+1491564us) ts_tablet_manager.cc:1151] Deleting tablet f909d82e0136470a810e1063d6769868
1211 08:57:36.466123 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.466326 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1490951}
I20251211 08:57:36.467314 14718 raft_consensus.cc:2243] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:36.467788 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 145) took 1375 ms. Trace:
I20251211 08:57:36.467913 14718 pending_rounds.cc:70] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:36.468118 14718 pending_rounds.cc:77] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256396170932224 op_type: NO_OP noop_request { }
I20251211 08:57:36.468052 14904 rpcz_store.cc:276] 1211 08:57:35.092150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.092258 (+ 108us) service_pool.cc:225] Handling call
1211 08:57:36.467538 (+1375280us) ts_tablet_manager.cc:1151] Deleting tablet d235bade012540e1bb1c0dd871eff1bb
1211 08:57:36.467607 (+ 69us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.467746 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1372646}
I20251211 08:57:36.468406 14718 raft_consensus.cc:2889] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:36.468683 14718 raft_consensus.cc:2272] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:36.468436 15030 raft_consensus.cc:359] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.469058 15030 raft_consensus.cc:385] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:36.469296 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 144) took 1377 ms. Trace:
I20251211 08:57:36.469302 15030 raft_consensus.cc:740] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:36.470510 14722 tablet_bootstrap.cc:492] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:36.469560 14904 rpcz_store.cc:276] 1211 08:57:35.091820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.130927 (+ 39107us) service_pool.cc:225] Handling call
1211 08:57:36.469092 (+1338165us) ts_tablet_manager.cc:1151] Deleting tablet 7b9b338a3f544ccb813247250b0f0eb7
1211 08:57:36.469126 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.469255 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1337578}
I20251211 08:57:36.474782 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 8ad600554c0946e0bcb0da6a42be2ffd
I20251211 08:57:36.475950 14008 catalog_manager.cc:6724] ReplaceTablet: tablet 8ad600554c0946e0bcb0da6a42be2ffd deleted and replaced by tablet 6874fb018bc844b3aeeab7db8fc53e9a
I20251211 08:57:36.475968 14718 ts_tablet_manager.cc:1916] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:36.477180 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 148) took 1155 ms. Trace:
I20251211 08:57:36.477557 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ad600554c0946e0bcb0da6a42be2ffd with delete_type TABLET_DATA_DELETED (replaced by tablet 6874fb018bc844b3aeeab7db8fc53e9a) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:36.477450 14904 rpcz_store.cc:276] 1211 08:57:35.322107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.322327 (+ 220us) service_pool.cc:225] Handling call
1211 08:57:36.476888 (+1154561us) ts_tablet_manager.cc:1151] Deleting tablet b8d753358f20476ba23d7d7b72e26e74
1211 08:57:36.476926 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.477137 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138036}
I20251211 08:57:36.478142 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ad600554c0946e0bcb0da6a42be2ffd with delete_type TABLET_DATA_DELETED (replaced by tablet 6874fb018bc844b3aeeab7db8fc53e9a) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:36.479691 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 125) took 2550 ms. Trace:
I20251211 08:57:36.487591 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 151) took 1074 ms. Trace:
I20251211 08:57:36.487881 14904 rpcz_store.cc:276] 1211 08:57:35.412913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.413048 (+ 135us) service_pool.cc:225] Handling call
1211 08:57:36.487363 (+1074315us) ts_tablet_manager.cc:1151] Deleting tablet 25805a95349d49f58280a30e6876929d
1211 08:57:36.487403 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.487527 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":996409}
I20251211 08:57:36.494261 15228 consensus_queue.cc:1048] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:36.500718 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 152) took 1050 ms. Trace:
I20251211 08:57:36.501039 14904 rpcz_store.cc:276] 1211 08:57:35.450523 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.450688 (+ 165us) service_pool.cc:225] Handling call
1211 08:57:36.499055 (+1048367us) ts_tablet_manager.cc:1151] Deleting tablet db302282bc3d4d8cb56d1cbb2f73c4fe
1211 08:57:36.499097 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.500647 (+ 1550us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008009}
I20251211 08:57:36.480034 14905 rpcz_store.cc:276] 1211 08:57:33.929568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:33.930127 (+ 559us) service_pool.cc:225] Handling call
1211 08:57:36.394559 (+2464432us) ts_tablet_manager.cc:1151] Deleting tablet caf537c461c04a81b66f268ee98dc64e
1211 08:57:36.394596 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.424537 (+ 29941us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:36.453503 (+ 28966us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:36.479617 (+ 26114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367629}
I20251211 08:57:36.502861 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet caf537c461c04a81b66f268ee98dc64e (table default.Table_ac11e9583ef64c1bbae0b2940213b819 [id=8c183fcd39ae4bbeb0a05bfdd42e755e]) successfully deleted
I20251211 08:57:36.506309 14722 tablet_bootstrap.cc:654] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:36.504292 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 129) took 2330 ms. Trace:
I20251211 08:57:36.506698 15213 tablet.cc:1722] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:36.508042 14905 rpcz_store.cc:276] 1211 08:57:34.174095 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.190663 (+ 16568us) service_pool.cc:225] Handling call
1211 08:57:36.503987 (+2313324us) ts_tablet_manager.cc:1151] Deleting tablet 3c5315f708c04322919c714dc915a483
1211 08:57:36.504028 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.504244 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2312640}
I20251211 08:57:36.469847 15030 consensus_queue.cc:260] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.513346 14905 tablet_replica.cc:333] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:36.514084 15030 ts_tablet_manager.cc:1434] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.218s user 0.004s sys 0.001s
I20251211 08:57:36.517248 15030 tablet_bootstrap.cc:492] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:36.517489 14905 raft_consensus.cc:2243] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:36.517927 14905 raft_consensus.cc:2272] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:36.518909 15214 tablet.cc:1722] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251211 08:57:36.523638 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet a6570281e1064c79b6e2297723f28658 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6570281e1064c79b6e2297723f28658 already in progress: creating tablet
I20251211 08:57:36.503098 14718 ts_tablet_manager.cc:1929] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251211 08:57:36.524358 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet f909d82e0136470a810e1063d6769868 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f909d82e0136470a810e1063d6769868 already in progress: creating tablet
I20251211 08:57:36.524592 14718 log.cc:1199] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/1976d584cc0a450ab809ead258a251e2
W20251211 08:57:36.524839 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet d235bade012540e1bb1c0dd871eff1bb because the tablet was not found. No further retry: Not found: Tablet not found: d235bade012540e1bb1c0dd871eff1bb
I20251211 08:57:36.525259 14718 ts_tablet_manager.cc:1950] T 1976d584cc0a450ab809ead258a251e2 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:36.527889 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 1976d584cc0a450ab809ead258a251e2 (table default.Table_81eee4b23e9045848867df8a0f511ae5 [id=72d8518a3dbf4dfdb28b1f422e44d414]) successfully deleted
W20251211 08:57:36.532482 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 2016208a33dc4a5cadd7ced8e85ccc65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2016208a33dc4a5cadd7ced8e85ccc65 already in progress: creating tablet
W20251211 08:57:36.534358 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b5a4b6b1b28406c82e2acd9356b60d4 already in progress: creating tablet
I20251211 08:57:36.535079 14714 tablet_bootstrap.cc:492] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:36.535459 14714 ts_tablet_manager.cc:1403] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.185s user 0.022s sys 0.006s
W20251211 08:57:36.536006 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 2cd6e4352cfc41e497dc58d158e24ff2 because the tablet was not found. No further retry: Not found: Tablet not found: 2cd6e4352cfc41e497dc58d158e24ff2
I20251211 08:57:36.536522 15122 raft_consensus.cc:493] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:36.537078 14718 tablet_replica.cc:333] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:36.536890 15122 raft_consensus.cc:515] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.537436 14714 raft_consensus.cc:359] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.537987 14714 raft_consensus.cc:385] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:36.538199 14714 raft_consensus.cc:740] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:36.538821 14718 raft_consensus.cc:2243] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:36.539743 14718 raft_consensus.cc:2272] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:36.540371 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55408f39a414e2caf1db48e6f29eca7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:36.538681 14714 consensus_queue.cc:260] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:36.541535 14382 leader_election.cc:343] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:36.541981 15122 leader_election.cc:290] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:36.542598 14714 ts_tablet_manager.cc:1434] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251211 08:57:36.544888 14718 ts_tablet_manager.cc:1916] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:36.546248 14714 tablet_bootstrap.cc:492] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:36.559688 14714 tablet_bootstrap.cc:654] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:36.571599 14718 ts_tablet_manager.cc:1929] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:36.574527 14718 log.cc:1199] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/a4e307e4d1994af0b99a88b3b06b73a4
I20251211 08:57:36.575488 14718 ts_tablet_manager.cc:1950] T a4e307e4d1994af0b99a88b3b06b73a4 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:36.522624 15030 tablet_bootstrap.cc:654] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:36.579013 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet a4e307e4d1994af0b99a88b3b06b73a4 (table default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]) successfully deleted
W20251211 08:57:36.581889 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 8ad600554c0946e0bcb0da6a42be2ffd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ad600554c0946e0bcb0da6a42be2ffd already in progress: creating tablet
I20251211 08:57:36.584067 14714 tablet_bootstrap.cc:492] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:36.584508 14714 ts_tablet_manager.cc:1403] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.000s
I20251211 08:57:36.590488 14905 ts_tablet_manager.cc:1916] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:36.526702 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 7b9b338a3f544ccb813247250b0f0eb7 because the tablet was not found. No further retry: Not found: Tablet not found: 7b9b338a3f544ccb813247250b0f0eb7
W20251211 08:57:36.602309 14000 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 75655.7us
W20251211 08:57:36.602825 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet b8d753358f20476ba23d7d7b72e26e74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8d753358f20476ba23d7d7b72e26e74 already in progress: creating tablet
W20251211 08:57:36.603482 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 25805a95349d49f58280a30e6876929d because the tablet was not found. No further retry: Not found: Tablet not found: 25805a95349d49f58280a30e6876929d
W20251211 08:57:36.604113 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet db302282bc3d4d8cb56d1cbb2f73c4fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db302282bc3d4d8cb56d1cbb2f73c4fe already in progress: creating tablet
W20251211 08:57:36.604714 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet c53035e83710403bac3a2b401c8c97ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c53035e83710403bac3a2b401c8c97ad already in progress: creating tablet
W20251211 08:57:36.605409 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 42720f5f5bf445f2b72e513ad4a6d95f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42720f5f5bf445f2b72e513ad4a6d95f already in progress: creating tablet
I20251211 08:57:36.598896 14714 raft_consensus.cc:359] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.606104 14714 raft_consensus.cc:385] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:36.606396 14714 raft_consensus.cc:740] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
W20251211 08:57:36.608255 15240 log.cc:927] Time spent T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063: Append to log took a long time: real 0.077s user 0.003s sys 0.000s
I20251211 08:57:36.611227 14714 consensus_queue.cc:260] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.603768 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ad600554c0946e0bcb0da6a42be2ffd with delete_type TABLET_DATA_DELETED (replaced by tablet 6874fb018bc844b3aeeab7db8fc53e9a) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:36.613150 14714 ts_tablet_manager.cc:1434] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20251211 08:57:36.614320 14714 tablet_bootstrap.cc:492] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:36.621644 14905 ts_tablet_manager.cc:1929] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:36.621963 14905 log.cc:1199] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/a0f9c321e673433498a9059dd2e2efe6
I20251211 08:57:36.622520 14904 tablet_replica.cc:333] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:36.622603 14905 ts_tablet_manager.cc:1950] T a0f9c321e673433498a9059dd2e2efe6 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:36.623210 14904 raft_consensus.cc:2243] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:36.623634 14904 raft_consensus.cc:2272] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:36.631544 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55408f39a414e2caf1db48e6f29eca7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:36.632069 14297 raft_consensus.cc:2468] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:36.633594 14385 leader_election.cc:304] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:36.635164 15122 raft_consensus.cc:2804] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:36.635474 15122 raft_consensus.cc:493] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:36.635779 15122 raft_consensus.cc:3060] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:36.642712 14714 tablet_bootstrap.cc:654] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:36.637861 15030 tablet_bootstrap.cc:492] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
W20251211 08:57:36.645922 14542 consensus_peers.cc:597] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cd6e4352cfc41e497dc58d158e24ff2. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:36.655890 15030 ts_tablet_manager.cc:1403] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.139s user 0.010s sys 0.005s
I20251211 08:57:36.658944 15030 raft_consensus.cc:359] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.660228 15030 raft_consensus.cc:385] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:36.660475 15030 raft_consensus.cc:740] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:36.660987 14904 ts_tablet_manager.cc:1916] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:36.661083 15030 consensus_queue.cc:260] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.667132 15030 ts_tablet_manager.cc:1434] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.011s user 0.001s sys 0.005s
I20251211 08:57:36.668160 15030 tablet_bootstrap.cc:492] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
W20251211 08:57:36.675978 14544 consensus_peers.cc:597] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:36.677461 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 130) took 2418 ms. Trace:
I20251211 08:57:36.677740 14905 rpcz_store.cc:276] 1211 08:57:34.259566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.259676 (+ 110us) service_pool.cc:225] Handling call
1211 08:57:36.513225 (+2253549us) ts_tablet_manager.cc:1151] Deleting tablet a0f9c321e673433498a9059dd2e2efe6
1211 08:57:36.513257 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.589379 (+ 76122us) spinlock_profiling.cc:243] Waited 56.3 ms on lock 0x7b24000025b8. stack: 00007f0c13f80083 00007f0c13f7ff80 00007f0c13af5574 0000000000508744 00000000005086d9 00007f0c13df2781 00007f0c13e06d5e 00007f0c13e04b55 00007f0c13e92638 00007f0c13e9287d 00007f0c13e91ec0 0000000000503085 0000000000503029 00000000005ad3a8 00007f0c1ae5ebfa 00007f0c1ae5ebbe
1211 08:57:36.604846 (+ 15467us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:36.621435 (+ 16589us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:36.677412 (+ 55977us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3437,"spinlock_wait_cycles":129538816,"tablet-delete.queue_time_us":2252981}
I20251211 08:57:36.678114 15122 raft_consensus.cc:515] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.678941 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet a0f9c321e673433498a9059dd2e2efe6 (table default.Table_d429fb0e546d4f3d886de3a04af241cf [id=649471fc3b2d4487b83e781f99ce77f5]) successfully deleted
I20251211 08:57:36.679849 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 131) took 2273 ms. Trace:
I20251211 08:57:36.680126 14905 rpcz_store.cc:276] 1211 08:57:34.406588 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.406696 (+ 108us) service_pool.cc:225] Handling call
1211 08:57:36.679555 (+2272859us) ts_tablet_manager.cc:1151] Deleting tablet 5daba165262e41af8d3481d3b4929092
1211 08:57:36.679599 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.679801 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2272358}
I20251211 08:57:36.680440 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55408f39a414e2caf1db48e6f29eca7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:36.680920 14297 raft_consensus.cc:3060] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:36.681200 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 5daba165262e41af8d3481d3b4929092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5daba165262e41af8d3481d3b4929092 already in progress: creating tablet
I20251211 08:57:36.687084 14297 raft_consensus.cc:2468] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:36.688083 14385 leader_election.cc:304] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:36.688818 15193 raft_consensus.cc:2804] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:36.691105 14905 tablet_replica.cc:333] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:36.691779 14905 raft_consensus.cc:2243] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:36.691787 15193 raft_consensus.cc:697] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:36.692530 15193 consensus_queue.cc:237] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.692164 14905 raft_consensus.cc:2272] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:36.698276 15192 raft_consensus.cc:493] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:36.698678 15192 raft_consensus.cc:515] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.700412 15192 leader_election.cc:290] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:36.700999 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afe339c4cc95490683618e18a3da252b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:36.703943 14385 leader_election.cc:343] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:36.708343 14714 tablet_bootstrap.cc:492] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:36.708717 14714 ts_tablet_manager.cc:1403] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.095s user 0.011s sys 0.000s
I20251211 08:57:36.711719 14905 ts_tablet_manager.cc:1916] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:36.711836 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afe339c4cc95490683618e18a3da252b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:36.712301 14605 raft_consensus.cc:2468] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:36.713116 14382 leader_election.cc:304] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:36.713977 15192 raft_consensus.cc:2804] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:36.714239 15192 raft_consensus.cc:493] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:36.714473 15192 raft_consensus.cc:3060] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:36.715842 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet ad903727f79a4180adf047c92d981c7e
I20251211 08:57:36.717090 15030 tablet_bootstrap.cc:654] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:36.717304 14013 catalog_manager.cc:6724] ReplaceTablet: tablet ad903727f79a4180adf047c92d981c7e deleted and replaced by tablet 9cc3373962574253868ababbbe78e57d
I20251211 08:57:36.718633 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet ad903727f79a4180adf047c92d981c7e with delete_type TABLET_DATA_DELETED (replaced by tablet 9cc3373962574253868ababbbe78e57d) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:36.719269 14718 tablet_replica.cc:333] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:36.719969 14718 raft_consensus.cc:2243] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:36.720387 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet ad903727f79a4180adf047c92d981c7e with delete_type TABLET_DATA_DELETED (replaced by tablet 9cc3373962574253868ababbbe78e57d) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:36.720621 14718 raft_consensus.cc:2272] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:36.722168 15192 raft_consensus.cc:515] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.726824 14718 ts_tablet_manager.cc:1916] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:36.732465 15122 leader_election.cc:290] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:36.732595 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55408f39a414e2caf1db48e6f29eca7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
W20251211 08:57:36.733691 14382 leader_election.cc:343] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:36.736034 14714 raft_consensus.cc:359] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.736570 14714 raft_consensus.cc:385] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:36.739279 14718 ts_tablet_manager.cc:1929] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:36.739576 14718 log.cc:1199] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/ad903727f79a4180adf047c92d981c7e
I20251211 08:57:36.740262 14718 ts_tablet_manager.cc:1950] T ad903727f79a4180adf047c92d981c7e P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:36.742264 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet ad903727f79a4180adf047c92d981c7e (table default.Table_dda8864ad51542599828719c4cb52e5b [id=2c2c410df07d4e4cbc60e08dce22efe9]) successfully deleted
I20251211 08:57:36.736847 14714 raft_consensus.cc:740] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:36.745639 14714 consensus_queue.cc:260] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:36.755367 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 174fefd70bcc41418a77a5716f97cd5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 174fefd70bcc41418a77a5716f97cd5f already in progress: creating tablet
I20251211 08:57:36.759481 15030 tablet_bootstrap.cc:492] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:36.759876 15030 ts_tablet_manager.cc:1403] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.092s user 0.007s sys 0.004s
I20251211 08:57:36.762058 15030 raft_consensus.cc:359] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.765933 15213 tablet.cc:1722] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:36.766935 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_cf5e7d3231db4b53b9ba619336f5bb19" } new_table_name: "default.Table_e444843f69064fbb9c2a6d0671fd9a2a" modify_external_catalogs: true
I20251211 08:57:36.770551 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet ad903727f79a4180adf047c92d981c7e with delete_type TABLET_DATA_DELETED (replaced by tablet 9cc3373962574253868ababbbe78e57d) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:36.762614 15030 raft_consensus.cc:385] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:36.775081 15030 raft_consensus.cc:740] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:36.777472 15030 consensus_queue.cc:260] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.784965 15214 tablet.cc:1722] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:36.800120 15192 leader_election.cc:290] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:36.800168 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afe339c4cc95490683618e18a3da252b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:36.800625 14608 raft_consensus.cc:3060] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:36.801550 14714 ts_tablet_manager.cc:1434] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.093s user 0.006s sys 0.000s
I20251211 08:57:36.802428 14714 tablet_bootstrap.cc:492] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:36.806365 15213 tablet.cc:1722] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:36.808665 14608 raft_consensus.cc:2468] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:36.809821 14382 leader_election.cc:304] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:36.814697 15193 raft_consensus.cc:2804] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:36.815106 15193 raft_consensus.cc:697] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:36.815793 15193 consensus_queue.cc:237] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.823393 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ccad8285d34949138091df8e2fafd435"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:36.824733 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccad8285d34949138091df8e2fafd435 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:36.826728 14722 tablet_bootstrap.cc:492] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:36.827133 14722 ts_tablet_manager.cc:1403] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.357s user 0.013s sys 0.008s
I20251211 08:57:36.832832 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_44453967d1924a139aaf13c5030ace59" } modify_external_catalogs: true
I20251211 08:57:36.833395 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_44453967d1924a139aaf13c5030ace59" } modify_external_catalogs: true
I20251211 08:57:36.840376 14905 ts_tablet_manager.cc:1929] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:36.840828 14905 log.cc:1199] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/1976d584cc0a450ab809ead258a251e2
I20251211 08:57:36.843742 15030 ts_tablet_manager.cc:1434] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.084s user 0.006s sys 0.000s
I20251211 08:57:36.844612 15030 tablet_bootstrap.cc:492] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:36.850378 14714 tablet_bootstrap.cc:654] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:36.855129 15263 tablet.cc:1722] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:36.856853 15214 tablet.cc:1722] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251211 08:57:36.872386 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 126167us
I20251211 08:57:36.873732 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afe339c4cc95490683618e18a3da252b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:36.876919 14385 leader_election.cc:343] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:36.880573 15030 tablet_bootstrap.cc:654] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:36.891484 14905 ts_tablet_manager.cc:1950] T 1976d584cc0a450ab809ead258a251e2 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:36.893375 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 132) took 2126 ms. Trace:
I20251211 08:57:36.893684 14905 rpcz_store.cc:276] 1211 08:57:34.766497 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.766629 (+ 132us) service_pool.cc:225] Handling call
1211 08:57:36.690959 (+1924330us) ts_tablet_manager.cc:1151] Deleting tablet 1976d584cc0a450ab809ead258a251e2
1211 08:57:36.690998 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.764883 (+ 73885us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:36.840104 (+ 75221us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:36.893314 (+ 53210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1923800}
I20251211 08:57:36.894809 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 1976d584cc0a450ab809ead258a251e2 (table default.Table_81eee4b23e9045848867df8a0f511ae5 [id=72d8518a3dbf4dfdb28b1f422e44d414]) successfully deleted
I20251211 08:57:36.867238 15258 tablet.cc:1722] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251211 08:57:36.914124 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5b462cf21e3420c8e323a4ac4f04abe consensus state has no leader
I20251211 08:57:36.913848 15226 tablet.cc:1722] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:36.920161 15260 tablet.cc:1722] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:36.898185 14722 raft_consensus.cc:359] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.926103 14722 raft_consensus.cc:385] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:36.926378 14722 raft_consensus.cc:740] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:36.927062 14722 consensus_queue.cc:260] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.928179 15265 tablet.cc:1722] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:36.936544 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 133) took 2116 ms. Trace:
I20251211 08:57:36.936872 14905 rpcz_store.cc:276] 1211 08:57:34.819912 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.820043 (+ 131us) service_pool.cc:225] Handling call
1211 08:57:36.936203 (+2116160us) ts_tablet_manager.cc:1151] Deleting tablet a6570281e1064c79b6e2297723f28658
1211 08:57:36.936242 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.936494 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2009466}
I20251211 08:57:36.937251 14904 ts_tablet_manager.cc:1929] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:36.937582 14904 log.cc:1199] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/2cd6e4352cfc41e497dc58d158e24ff2
W20251211 08:57:36.938045 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet a6570281e1064c79b6e2297723f28658 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6570281e1064c79b6e2297723f28658 already in progress: creating tablet
I20251211 08:57:36.944540 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 136) took 1968 ms. Trace:
I20251211 08:57:36.944904 14905 rpcz_store.cc:276] 1211 08:57:34.976353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:34.976469 (+ 116us) service_pool.cc:225] Handling call
1211 08:57:36.944227 (+1967758us) ts_tablet_manager.cc:1151] Deleting tablet f909d82e0136470a810e1063d6769868
1211 08:57:36.944267 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.944491 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1967242}
W20251211 08:57:36.946245 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet f909d82e0136470a810e1063d6769868 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f909d82e0136470a810e1063d6769868 already in progress: creating tablet
I20251211 08:57:36.951359 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 138) took 1870 ms. Trace:
I20251211 08:57:36.953800 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d86a1b65094448658217d68127970c1a" } new_table_name: "default.Table_3e1d63cec2a5456788ac674e54f4669f" modify_external_catalogs: true
I20251211 08:57:36.959416 14905 rpcz_store.cc:276] 1211 08:57:35.081014 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.082891 (+ 1877us) service_pool.cc:225] Handling call
1211 08:57:36.951147 (+1868256us) ts_tablet_manager.cc:1151] Deleting tablet a9524aa1503549bd820918d1c025aa2e
1211 08:57:36.951181 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.951314 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1867740}
W20251211 08:57:36.960573 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet a9524aa1503549bd820918d1c025aa2e because the tablet was not found. No further retry: Not found: Tablet not found: a9524aa1503549bd820918d1c025aa2e
I20251211 08:57:36.961537 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 139) took 1880 ms. Trace:
I20251211 08:57:36.956329 15030 tablet_bootstrap.cc:492] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:36.962219 15030 ts_tablet_manager.cc:1403] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.118s user 0.005s sys 0.004s
I20251211 08:57:36.962258 14013 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet ee63c510c4bd402388b961c3c19c8303
I20251211 08:57:36.964483 15030 raft_consensus.cc:359] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:36.965142 15030 raft_consensus.cc:385] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:36.965428 15030 raft_consensus.cc:740] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:36.966501 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet 996c85729acd4418be1763b73cbf7d2d from {username='slave'} at 127.0.0.1:55614
W20251211 08:57:36.977183 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 552de4c6d030418181f0400714d9ec31 has no consensus state
I20251211 08:57:36.991714 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 996c85729acd4418be1763b73cbf7d2d
I20251211 08:57:36.993744 14013 catalog_manager.cc:6724] ReplaceTablet: tablet 996c85729acd4418be1763b73cbf7d2d deleted and replaced by tablet ac6be204692a44fab2db551e79237299
I20251211 08:57:36.994565 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 996c85729acd4418be1763b73cbf7d2d with delete_type TABLET_DATA_DELETED (replaced by tablet ac6be204692a44fab2db551e79237299) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:36.992914 14905 rpcz_store.cc:276] 1211 08:57:35.081299 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.083504 (+ 2205us) service_pool.cc:225] Handling call
1211 08:57:36.961300 (+1877796us) ts_tablet_manager.cc:1151] Deleting tablet 15f7bcd680064b9f8fe51aa05acce6fb
1211 08:57:36.961335 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:36.961495 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1877348}
W20251211 08:57:36.995824 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 15f7bcd680064b9f8fe51aa05acce6fb because the tablet was not found. No further retry: Not found: Tablet not found: 15f7bcd680064b9f8fe51aa05acce6fb
I20251211 08:57:36.998842 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 996c85729acd4418be1763b73cbf7d2d with delete_type TABLET_DATA_DELETED (replaced by tablet ac6be204692a44fab2db551e79237299) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:36.999450 14718 tablet_replica.cc:333] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:37.000134 14718 raft_consensus.cc:2243] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:37.010951 14718 raft_consensus.cc:2272] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:37.012643 14718 ts_tablet_manager.cc:1916] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:37.014294 15122 consensus_queue.cc:1048] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:37.015535 15182 consensus_queue.cc:1048] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:36.938251 14904 ts_tablet_manager.cc:1950] T 2cd6e4352cfc41e497dc58d158e24ff2 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:57:37.020814 14385 consensus_peers.cc:597] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:37.021176 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 140) took 1939 ms. Trace:
I20251211 08:57:37.021462 14905 rpcz_store.cc:276] 1211 08:57:35.081588 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.085858 (+ 4270us) service_pool.cc:225] Handling call
1211 08:57:37.020938 (+1935080us) ts_tablet_manager.cc:1151] Deleting tablet 25805a95349d49f58280a30e6876929d
1211 08:57:37.020977 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.021127 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1934616}
W20251211 08:57:37.022517 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 25805a95349d49f58280a30e6876929d because the tablet was not found. No further retry: Not found: Tablet not found: 25805a95349d49f58280a30e6876929d
I20251211 08:57:37.023619 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 141) took 1941 ms. Trace:
I20251211 08:57:36.993602 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet 996c85729acd4418be1763b73cbf7d2d with delete_type TABLET_DATA_DELETED (replaced by tablet ac6be204692a44fab2db551e79237299) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:37.023891 14905 rpcz_store.cc:276] 1211 08:57:35.081957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.086399 (+ 4442us) service_pool.cc:225] Handling call
1211 08:57:37.023237 (+1936838us) ts_tablet_manager.cc:1151] Deleting tablet 1853070f81c34b52b8c74a7402e6dad3
1211 08:57:37.023276 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.023448 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1936365}
I20251211 08:57:36.952284 14722 ts_tablet_manager.cc:1434] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.125s user 0.005s sys 0.000s
W20251211 08:57:37.024856 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 1853070f81c34b52b8c74a7402e6dad3 because the tablet was not found. No further retry: Not found: Tablet not found: 1853070f81c34b52b8c74a7402e6dad3
I20251211 08:57:37.026181 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 143) took 1938 ms. Trace:
I20251211 08:57:36.966095 15030 consensus_queue.cc:260] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.031884 14722 tablet_bootstrap.cc:492] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:37.033859 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 2cd6e4352cfc41e497dc58d158e24ff2 (table default.Table_9c85250afba741a79335663257915c82 [id=48f11a87431349f8a3caeb458f06869d]) successfully deleted
I20251211 08:57:37.037098 14904 tablet_replica.cc:333] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:37.038641 14904 raft_consensus.cc:2243] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:37.039163 14904 raft_consensus.cc:2272] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:37.040910 15030 ts_tablet_manager.cc:1434] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.078s user 0.001s sys 0.003s
I20251211 08:57:37.041100 14904 ts_tablet_manager.cc:1916] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:37.043416 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a72bb97d554348b6859583e04f29cf9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:37.044189 15030 tablet_bootstrap.cc:492] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
W20251211 08:57:37.044883 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a72bb97d554348b6859583e04f29cf9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:37.053948 14714 tablet_bootstrap.cc:492] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:37.054376 14714 ts_tablet_manager.cc:1403] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.252s user 0.011s sys 0.003s
I20251211 08:57:37.026520 14905 rpcz_store.cc:276] 1211 08:57:35.087307 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.087413 (+ 106us) service_pool.cc:225] Handling call
1211 08:57:37.025935 (+1938522us) ts_tablet_manager.cc:1151] Deleting tablet 292b641df22f478d8849ff6148ecf951
1211 08:57:37.025970 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.026139 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1938037}
I20251211 08:57:37.068313 15237 raft_consensus.cc:493] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.069648 15237 raft_consensus.cc:515] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.071848 15237 leader_election.cc:290] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:37.073419 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "383716a1f1f847ccbea542eca04790c9" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:37.074798 14714 raft_consensus.cc:359] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.075337 14714 raft_consensus.cc:385] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.075554 14714 raft_consensus.cc:740] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.075484 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "383716a1f1f847ccbea542eca04790c9" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:37.076099 14714 consensus_queue.cc:260] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:37.076996 14544 leader_election.cc:343] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:37.078028 14542 leader_election.cc:343] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.078366 14542 leader_election.cc:304] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:37.079689 15237 raft_consensus.cc:2749] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:37.080809 14722 tablet_bootstrap.cc:654] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.094635 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_1ce47a525fa9427198a4b8c589d662c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:37.096012 14714 ts_tablet_manager.cc:1434] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.041s user 0.001s sys 0.003s
I20251211 08:57:37.097021 14718 ts_tablet_manager.cc:1929] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:37.098623 14718 log.cc:1199] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/996c85729acd4418be1763b73cbf7d2d
I20251211 08:57:37.101631 15237 raft_consensus.cc:493] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.102100 14718 ts_tablet_manager.cc:1950] T 996c85729acd4418be1763b73cbf7d2d P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:37.102041 15237 raft_consensus.cc:515] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.103472 14714 tablet_bootstrap.cc:492] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:37.106472 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6c4d4c99b3b496cab125e13153349cb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:37.107079 14299 raft_consensus.cc:2468] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:37.108227 14544 leader_election.cc:304] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:37.108990 15152 raft_consensus.cc:2804] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:37.109282 15152 raft_consensus.cc:493] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:37.109586 15152 raft_consensus.cc:3060] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:37.109894 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 996c85729acd4418be1763b73cbf7d2d (table default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]) successfully deleted
I20251211 08:57:37.112350 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6c4d4c99b3b496cab125e13153349cb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:37.112872 14446 raft_consensus.cc:2468] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:37.115731 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_fee4bd06cd7e4df08fa4bafdd291a3d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:37.118314 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_90208748f83c43e9b2bf122c5425257e" } new_table_name: "default.Table_cdd8cc77899d40369a8e85dcff2587e2" modify_external_catalogs: true
I20251211 08:57:37.121347 15237 leader_election.cc:290] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:37.122429 15152 raft_consensus.cc:515] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.124073 14714 tablet_bootstrap.cc:654] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.126462 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_e71106e05fbc47248f2b5f71482c2231"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:37.128410 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6c4d4c99b3b496cab125e13153349cb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:37.128944 14299 raft_consensus.cc:3060] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:37.132501 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb consensus state has no leader
I20251211 08:57:37.138415 14722 tablet_bootstrap.cc:492] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:37.138839 14722 ts_tablet_manager.cc:1403] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.107s user 0.009s sys 0.001s
I20251211 08:57:37.141525 14722 raft_consensus.cc:359] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.142073 14722 raft_consensus.cc:385] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.142297 14722 raft_consensus.cc:740] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.144479 14714 tablet_bootstrap.cc:492] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:37.144819 14714 ts_tablet_manager.cc:1403] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.002s
I20251211 08:57:37.082629 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 145) took 1966 ms. Trace:
I20251211 08:57:37.146968 14714 raft_consensus.cc:359] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.147567 14714 raft_consensus.cc:385] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.147826 14714 raft_consensus.cc:740] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.148329 14714 consensus_queue.cc:260] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.153424 14714 ts_tablet_manager.cc:1434] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20251211 08:57:37.154289 14714 tablet_bootstrap.cc:492] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:37.165514 14009 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a72bb97d554348b6859583e04f29cf9f" } modify_external_catalogs: true
I20251211 08:57:37.166007 14009 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a72bb97d554348b6859583e04f29cf9f" } modify_external_catalogs: true
I20251211 08:57:37.167167 14714 tablet_bootstrap.cc:654] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.180370 14009 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet b8152bd279454f8cbe22d10ebb7c5c7b
I20251211 08:57:37.145294 14905 rpcz_store.cc:276] 1211 08:57:35.116198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.117178 (+ 980us) service_pool.cc:225] Handling call
1211 08:57:37.082092 (+1964914us) ts_tablet_manager.cc:1151] Deleting tablet 60b23163906447c3877db631968f5623
1211 08:57:37.082163 (+ 71us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.082420 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1960831}
I20251211 08:57:37.185627 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8363e9c4cd624dbe8ba7edf7ac61591d" } new_table_name: "default.Table_ca17fcc98d9d495e81e4d8186a51deeb" modify_external_catalogs: true
I20251211 08:57:37.185632 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 144) took 2092 ms. Trace:
W20251211 08:57:37.081605 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 292b641df22f478d8849ff6148ecf951 because the tablet was not found. No further retry: Not found: Tablet not found: 292b641df22f478d8849ff6148ecf951
W20251211 08:57:37.188028 13997 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 106500us
W20251211 08:57:37.189814 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 60b23163906447c3877db631968f5623 because the tablet was not found. No further retry: Not found: Tablet not found: 60b23163906447c3877db631968f5623
W20251211 08:57:37.190521 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fee4bd06cd7e4df08fa4bafdd291a3d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:37.197579 14714 tablet_bootstrap.cc:492] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:37.197933 14714 ts_tablet_manager.cc:1403] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.000s
I20251211 08:57:37.199972 14714 raft_consensus.cc:359] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.200567 14714 raft_consensus.cc:385] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.200827 14714 raft_consensus.cc:740] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.202963 14714 consensus_queue.cc:260] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.204181 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6c4d4c99b3b496cab125e13153349cb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:37.203806 15260 tablet.cc:1722] T 3303dc521d694be6b386a35da1172ede P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:37.204723 14446 raft_consensus.cc:3060] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:37.204902 14714 ts_tablet_manager.cc:1434] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251211 08:57:37.207667 15237 raft_consensus.cc:493] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.208164 15237 raft_consensus.cc:515] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.210829 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fc87cb76024b5b8227dfe9939e1c89" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:37.211371 14449 raft_consensus.cc:2468] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:37.211748 14714 tablet_bootstrap.cc:492] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:37.212364 14542 leader_election.cc:304] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:37.217403 14714 tablet_bootstrap.cc:654] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.218371 15226 tablet.cc:1722] T 3303dc521d694be6b386a35da1172ede P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:37.219457 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fc87cb76024b5b8227dfe9939e1c89" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:37.219539 15237 leader_election.cc:290] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:37.220024 15237 raft_consensus.cc:2804] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:37.221359 15237 raft_consensus.cc:493] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:37.222164 15237 raft_consensus.cc:3060] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:37.226992 14544 leader_election.cc:343] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.227669 15193 raft_consensus.cc:493] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.228246 15193 raft_consensus.cc:515] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.230021 15193 leader_election.cc:290] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:37.231133 14722 consensus_queue.cc:260] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:37.233556 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ce47a525fa9427198a4b8c589d662c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:37.233556 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "383716a1f1f847ccbea542eca04790c9" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:37.234064 14605 raft_consensus.cc:2468] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:37.232228 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "383716a1f1f847ccbea542eca04790c9" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:37.236555 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb consensus state has no leader
W20251211 08:57:37.236600 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e71106e05fbc47248f2b5f71482c2231 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:37.233259 14446 raft_consensus.cc:2468] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:37.242970 15152 leader_election.cc:290] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:37.243268 14722 ts_tablet_manager.cc:1434] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.104s user 0.005s sys 0.001s
I20251211 08:57:37.235060 14382 leader_election.cc:304] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
W20251211 08:57:37.243881 14385 leader_election.cc:343] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.246917 15193 raft_consensus.cc:2804] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:37.247586 15193 raft_consensus.cc:493] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:37.248122 15193 raft_consensus.cc:3060] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:37.230852 15265 tablet.cc:1722] T 3303dc521d694be6b386a35da1172ede P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:37.251817 14722 tablet_bootstrap.cc:492] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:37.253083 15030 tablet_bootstrap.cc:654] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.186730 14905 rpcz_store.cc:276] 1211 08:57:35.115840 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.117219 (+ 1379us) service_pool.cc:225] Handling call
1211 08:57:37.184727 (+2067508us) ts_tablet_manager.cc:1151] Deleting tablet 426e26859b9c4966b8ec8ba3eab53a43
1211 08:57:37.184811 (+ 84us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.185466 (+ 655us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2062852}
W20251211 08:57:37.255354 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 426e26859b9c4966b8ec8ba3eab53a43 because the tablet was not found. No further retry: Not found: Tablet not found: 426e26859b9c4966b8ec8ba3eab53a43
I20251211 08:57:37.259378 15192 raft_consensus.cc:493] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.259804 15192 raft_consensus.cc:515] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.261636 15192 leader_election.cc:290] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:37.262805 14904 ts_tablet_manager.cc:1929] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:37.263144 14904 log.cc:1199] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/479d7fe06a284234aef420e3842f0eab
I20251211 08:57:37.263818 14904 ts_tablet_manager.cc:1950] T 479d7fe06a284234aef420e3842f0eab P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:37.266856 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf537c461c04a81b66f268ee98dc64e" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:37.267148 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6109f16f9985461fbb07b1d083bbb166"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:37.269162 14385 leader_election.cc:343] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.244366 14299 raft_consensus.cc:2468] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:37.274878 14905 tablet_replica.cc:333] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:37.275560 14905 raft_consensus.cc:2243] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
W20251211 08:57:37.275576 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b9ae463731d4828afc02039e0d161b5 has no consensus state
I20251211 08:57:37.276484 14905 raft_consensus.cc:2272] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:37.274626 14544 leader_election.cc:304] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:37.277972 14905 ts_tablet_manager.cc:1916] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:37.279976 14382 leader_election.cc:343] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: caf537c461c04a81b66f268ee98dc64e
I20251211 08:57:37.280332 14382 leader_election.cc:304] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:37.281507 15122 raft_consensus.cc:2749] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:37.281939 15122 consensus_queue.cc:1048] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:37.285818 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 73567.6us
I20251211 08:57:37.287142 15273 raft_consensus.cc:2804] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:37.288269 15273 raft_consensus.cc:697] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:37.289134 15273 consensus_queue.cc:237] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.282362 15237 raft_consensus.cc:515] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.317189 14007 master_service.cc:954] ReplaceTablet: received request to replace tablet afe339c4cc95490683618e18a3da252b from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:37.319139 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fc87cb76024b5b8227dfe9939e1c89" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:37.322512 14446 raft_consensus.cc:3060] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:37.330956 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6109f16f9985461fbb07b1d083bbb166 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:37.335191 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 165) took 1289 ms. Trace:
I20251211 08:57:37.335531 14904 rpcz_store.cc:276] 1211 08:57:36.045823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.073227 (+ 27404us) service_pool.cc:225] Handling call
1211 08:57:37.036970 (+963743us) ts_tablet_manager.cc:1151] Deleting tablet 479d7fe06a284234aef420e3842f0eab
1211 08:57:37.037004 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.146282 (+109278us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:37.262579 (+116297us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:37.335133 (+ 72554us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":963290}
I20251211 08:57:37.338305 14722 tablet_bootstrap.cc:654] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.338430 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 479d7fe06a284234aef420e3842f0eab (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]) successfully deleted
I20251211 08:57:37.343241 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fc87cb76024b5b8227dfe9939e1c89" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:37.344576 14544 leader_election.cc:343] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.351538 15237 leader_election.cc:290] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:37.355451 14446 raft_consensus.cc:2468] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:37.367551 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 168) took 1123 ms. Trace:
I20251211 08:57:37.367869 14904 rpcz_store.cc:276] 1211 08:57:36.244568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.262470 (+ 17902us) service_pool.cc:225] Handling call
1211 08:57:37.367211 (+1104741us) ts_tablet_manager.cc:1151] Deleting tablet 2016208a33dc4a5cadd7ced8e85ccc65
1211 08:57:37.367252 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.367490 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1104146}
W20251211 08:57:37.368975 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 2016208a33dc4a5cadd7ced8e85ccc65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2016208a33dc4a5cadd7ced8e85ccc65 already in progress: creating tablet
I20251211 08:57:37.371408 14542 leader_election.cc:304] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:37.363160 15193 raft_consensus.cc:515] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.373764 15193 leader_election.cc:290] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:37.375038 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "383716a1f1f847ccbea542eca04790c9" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:37.375873 15152 raft_consensus.cc:2804] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
W20251211 08:57:37.375967 14385 leader_election.cc:343] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.376525 15152 raft_consensus.cc:697] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
W20251211 08:57:37.378088 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b5a4b6b1b28406c82e2acd9356b60d4 already in progress: creating tablet
I20251211 08:57:37.382295 15030 tablet_bootstrap.cc:492] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:37.383523 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "383716a1f1f847ccbea542eca04790c9" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
W20251211 08:57:37.383984 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 8ad600554c0946e0bcb0da6a42be2ffd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ad600554c0946e0bcb0da6a42be2ffd already in progress: creating tablet
I20251211 08:57:37.384140 14607 raft_consensus.cc:3060] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:37.382822 15030 ts_tablet_manager.cc:1403] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.341s user 0.009s sys 0.004s
I20251211 08:57:37.387269 14904 tablet_replica.cc:333] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:37.388096 14904 raft_consensus.cc:2243] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:37.388677 14904 pending_rounds.cc:70] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
I20251211 08:57:37.388898 14904 pending_rounds.cc:77] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7231256397304582144 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "ad903727f79a4180adf047c92d981c7e" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_dda8864ad51542599828719c4cb52e5b" dest_uuid: "b276f77f87254a5d909c5ab940a39256" new_extra_config { } }
I20251211 08:57:37.390233 14904 raft_consensus.cc:2272] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:37.394223 14004 master_service.cc:954] ReplaceTablet: received request to replace tablet e6c4d4c99b3b496cab125e13153349cb from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:37.397603 15030 raft_consensus.cc:359] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.398535 15030 raft_consensus.cc:385] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.398870 15030 raft_consensus.cc:740] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.399641 15030 consensus_queue.cc:260] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.405440 15193 raft_consensus.cc:493] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.405897 15193 raft_consensus.cc:515] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.406852 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_150512d0e0b5469694652b5e2e39dabc" } new_table_name: "default.Table_dcb6d015c5c543689fc117f18bdff260" modify_external_catalogs: true
I20251211 08:57:37.408226 15193 leader_election.cc:290] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:37.408764 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8d753358f20476ba23d7d7b72e26e74" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:37.410049 14385 leader_election.cc:343] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.411636 14904 ts_tablet_manager.cc:1916] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:37.418759 14722 tablet_bootstrap.cc:492] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:37.419101 14722 ts_tablet_manager.cc:1403] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.167s user 0.009s sys 0.006s
I20251211 08:57:37.425567 14714 tablet_bootstrap.cc:492] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:37.411592 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8d753358f20476ba23d7d7b72e26e74" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:37.411913 14607 raft_consensus.cc:2468] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:37.425936 14714 ts_tablet_manager.cc:1403] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.214s user 0.050s sys 0.042s
I20251211 08:57:37.439905 14714 raft_consensus.cc:359] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.440510 14714 raft_consensus.cc:385] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.440793 14714 raft_consensus.cc:740] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.443107 14714 consensus_queue.cc:260] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.444276 14382 leader_election.cc:304] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:37.445103 14714 ts_tablet_manager.cc:1434] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20251211 08:57:37.445616 15083 raft_consensus.cc:2804] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:37.445997 15083 raft_consensus.cc:697] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:37.446643 15083 consensus_queue.cc:237] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.377368 15152 consensus_queue.cc:237] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.442960 14605 raft_consensus.cc:2468] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:37.449469 14714 tablet_bootstrap.cc:492] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:37.437430 14722 raft_consensus.cc:359] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.451815 14722 raft_consensus.cc:385] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.452067 14722 raft_consensus.cc:740] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.453847 14722 consensus_queue.cc:260] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.454579 14714 tablet_bootstrap.cc:654] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.459134 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_96d05b744df34836be68b4841e6c2a2b" } modify_external_catalogs: true
I20251211 08:57:37.459643 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_96d05b744df34836be68b4841e6c2a2b" } modify_external_catalogs: true
I20251211 08:57:37.462894 15030 ts_tablet_manager.cc:1434] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.078s user 0.002s sys 0.004s
I20251211 08:57:37.467327 15030 tablet_bootstrap.cc:492] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:37.470813 15083 raft_consensus.cc:493] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.471694 14382 leader_election.cc:304] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
W20251211 08:57:37.475739 14013 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00e5c4bf71b44abca66b0e94995c533e has no consensus state
I20251211 08:57:37.477720 15193 raft_consensus.cc:2804] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:37.478430 14722 ts_tablet_manager.cc:1434] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.059s user 0.005s sys 0.001s
I20251211 08:57:37.478907 15193 raft_consensus.cc:493] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:37.479748 15193 raft_consensus.cc:3060] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:37.474999 15083 raft_consensus.cc:515] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.483842 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_5a97657fb7304e73bddac0c22b7ee787" } new_table_name: "default.Table_b72b9116d4994b90ad29513202ff4217" modify_external_catalogs: true
I20251211 08:57:37.484653 14714 tablet_bootstrap.cc:492] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:37.485018 14714 ts_tablet_manager.cc:1403] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.000s
I20251211 08:57:37.487157 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "363fe7a2e1ac4fb4a1ec97dc2354d71a" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:37.487215 14714 raft_consensus.cc:359] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.487783 14714 raft_consensus.cc:385] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.488031 14714 raft_consensus.cc:740] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
W20251211 08:57:37.488328 14382 leader_election.cc:343] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.488607 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "363fe7a2e1ac4fb4a1ec97dc2354d71a" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:37.489269 15083 leader_election.cc:290] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:37.489686 14385 leader_election.cc:343] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.490067 14385 leader_election.cc:304] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:37.490665 15083 raft_consensus.cc:2749] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:37.495518 14905 ts_tablet_manager.cc:1929] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:37.488715 14714 consensus_queue.cc:260] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.498723 14905 log.cc:1199] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/2cd6e4352cfc41e497dc58d158e24ff2
I20251211 08:57:37.499657 14714 ts_tablet_manager.cc:1434] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20251211 08:57:37.503861 14714 tablet_bootstrap.cc:492] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:37.502458 14722 tablet_bootstrap.cc:492] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:37.501601 14605 raft_consensus.cc:3060] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:37.509377 14714 tablet_bootstrap.cc:654] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.514983 15083 consensus_queue.cc:1048] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:37.515583 14013 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd14de369e084828afb8a924c46c649b has no consensus state
I20251211 08:57:37.504001 15030 tablet_bootstrap.cc:654] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.499819 14905 ts_tablet_manager.cc:1950] T 2cd6e4352cfc41e497dc58d158e24ff2 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:37.523427 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_c0e87778556043e9ad8242434ab127c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:37.524813 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0e87778556043e9ad8242434ab127c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:37.529157 15228 raft_consensus.cc:493] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.529625 15228 raft_consensus.cc:515] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.532296 15228 leader_election.cc:290] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:37.532164 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "383716a1f1f847ccbea542eca04790c9" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:37.534113 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "383716a1f1f847ccbea542eca04790c9" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:37.534775 14607 raft_consensus.cc:2393] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:37.536741 14722 tablet_bootstrap.cc:654] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.546473 15228 raft_consensus.cc:493] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.546913 15228 raft_consensus.cc:515] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.549237 15228 leader_election.cc:290] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:37.549691 15193 raft_consensus.cc:515] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.551621 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 148) took 2316 ms. Trace:
I20251211 08:57:37.551946 14905 rpcz_store.cc:276] 1211 08:57:35.234797 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.290039 (+ 55242us) service_pool.cc:225] Handling call
1211 08:57:37.274716 (+1984677us) ts_tablet_manager.cc:1151] Deleting tablet 2cd6e4352cfc41e497dc58d158e24ff2
1211 08:57:37.274755 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.420866 (+146111us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:37.495312 (+ 74446us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:37.551546 (+ 56234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1984192}
W20251211 08:57:37.552690 14231 leader_election.cc:343] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: a4e307e4d1994af0b99a88b3b06b73a4
I20251211 08:57:37.553143 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 2cd6e4352cfc41e497dc58d158e24ff2 (table default.Table_9c85250afba741a79335663257915c82 [id=48f11a87431349f8a3caeb458f06869d]) successfully deleted
I20251211 08:57:37.554029 14905 tablet_replica.cc:333] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:37.554754 14905 raft_consensus.cc:2243] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:37.554701 14230 leader_election.cc:304] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:37.555755 15228 raft_consensus.cc:3060] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:37.558063 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8d753358f20476ba23d7d7b72e26e74" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:37.559146 14385 leader_election.cc:343] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.560330 14714 tablet_bootstrap.cc:492] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:37.560472 15193 leader_election.cc:290] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:37.560736 14714 ts_tablet_manager.cc:1403] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.009s
I20251211 08:57:37.562639 15228 raft_consensus.cc:2749] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:37.563122 14714 raft_consensus.cc:359] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.563648 14714 raft_consensus.cc:385] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.563868 14714 raft_consensus.cc:740] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.564348 14714 consensus_queue.cc:260] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.566073 14714 ts_tablet_manager.cc:1434] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251211 08:57:37.568189 14905 raft_consensus.cc:2272] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:37.568363 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8d753358f20476ba23d7d7b72e26e74" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:37.561610 15030 tablet_bootstrap.cc:492] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:37.569355 15030 ts_tablet_manager.cc:1403] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.106s user 0.004s sys 0.005s
I20251211 08:57:37.571646 15030 raft_consensus.cc:359] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.572264 15030 raft_consensus.cc:385] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.572481 15030 raft_consensus.cc:740] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.574822 15083 consensus_queue.cc:1048] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:37.584124 14714 tablet_bootstrap.cc:492] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:37.587292 14905 ts_tablet_manager.cc:1916] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:37.591285 14382 leader_election.cc:343] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.591658 14382 leader_election.cc:304] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:37.593925 15193 raft_consensus.cc:2749] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251211 08:57:37.596793 14714 tablet_bootstrap.cc:654] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.594595 15030 consensus_queue.cc:260] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:37.569976 14230 leader_election.cc:343] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: a4e307e4d1994af0b99a88b3b06b73a4
I20251211 08:57:37.573734 14904 ts_tablet_manager.cc:1929] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:37.599786 14230 leader_election.cc:304] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:37.599898 14904 log.cc:1199] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/ad903727f79a4180adf047c92d981c7e
W20251211 08:57:37.600596 14385 consensus_peers.cc:597] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:37.600709 15148 raft_consensus.cc:2749] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:37.600962 14904 ts_tablet_manager.cc:1950] T ad903727f79a4180adf047c92d981c7e P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:37.603993 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet ad903727f79a4180adf047c92d981c7e (table default.Table_dda8864ad51542599828719c4cb52e5b [id=2c2c410df07d4e4cbc60e08dce22efe9]) successfully deleted
I20251211 08:57:37.604849 14904 tablet_replica.cc:333] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:37.605571 14904 raft_consensus.cc:2243] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:37.608245 14904 raft_consensus.cc:2272] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:37.614068 14904 ts_tablet_manager.cc:1916] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:37.614064 15030 ts_tablet_manager.cc:1434] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.044s user 0.002s sys 0.003s
I20251211 08:57:37.616834 15030 tablet_bootstrap.cc:492] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:37.621773 15030 tablet_bootstrap.cc:654] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.624325 14722 tablet_bootstrap.cc:492] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:37.624739 14722 ts_tablet_manager.cc:1403] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.122s user 0.004s sys 0.009s
I20251211 08:57:37.626982 14722 raft_consensus.cc:359] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.627637 14722 raft_consensus.cc:385] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.627969 14714 tablet_bootstrap.cc:492] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:37.627910 14722 raft_consensus.cc:740] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.628360 14714 ts_tablet_manager.cc:1403] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.000s
I20251211 08:57:37.629127 14722 consensus_queue.cc:260] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.630442 14714 raft_consensus.cc:359] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.631055 14714 raft_consensus.cc:385] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.633383 14714 raft_consensus.cc:740] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.633888 14714 consensus_queue.cc:260] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.640239 14722 ts_tablet_manager.cc:1434] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20251211 08:57:37.647909 14714 ts_tablet_manager.cc:1434] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20251211 08:57:37.648707 14714 tablet_bootstrap.cc:492] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:37.653621 14714 tablet_bootstrap.cc:654] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.665021 14722 tablet_bootstrap.cc:492] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:37.671921 14905 ts_tablet_manager.cc:1929] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:37.672290 14905 log.cc:1199] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/b8d753358f20476ba23d7d7b72e26e74
I20251211 08:57:37.672942 14905 ts_tablet_manager.cc:1950] T b8d753358f20476ba23d7d7b72e26e74 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:37.684547 14714 tablet_bootstrap.cc:492] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:37.684994 14714 ts_tablet_manager.cc:1403] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.000s
I20251211 08:57:37.687858 14714 raft_consensus.cc:359] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.688673 14714 raft_consensus.cc:385] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.688948 14714 raft_consensus.cc:740] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.690542 14714 consensus_queue.cc:260] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.704196 14429 tablet_service.cc:1505] Processing CreateTablet for tablet 45c7d841bd3643a799a8b96e82d6d675 (DEFAULT_TABLE table=default.Table_4960eb1f338e47b3923b57ce8be73703 [id=f08b1a30da03451ba864cb8f8e6d199b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.705149 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45c7d841bd3643a799a8b96e82d6d675. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.708956 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 4cbd18e927c843e7b0aee7e0ccb1b06d (DEFAULT_TABLE table=default.Table_cb9f538930f64a52b0f18805535a8fc1 [id=a9e8baf63d354fdaab9fb2a0381ffbcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.710366 14269 tablet_service.cc:1505] Processing CreateTablet for tablet 45c7d841bd3643a799a8b96e82d6d675 (DEFAULT_TABLE table=default.Table_4960eb1f338e47b3923b57ce8be73703 [id=f08b1a30da03451ba864cb8f8e6d199b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.711323 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45c7d841bd3643a799a8b96e82d6d675. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.712843 14272 tablet_service.cc:1505] Processing CreateTablet for tablet c6d0fef50524476dbd47fc70e4bda319 (DEFAULT_TABLE table=default.Table_7919ce70cfc5400bb4f2515164c6b7a1 [id=3a336d5ad7004619951e901ad9bf39bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.698043 14714 ts_tablet_manager.cc:1434] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20251211 08:57:37.714061 14273 tablet_service.cc:1505] Processing CreateTablet for tablet da57b72207414512b9744b283bc869fe (DEFAULT_TABLE table=default.Table_2f325f51dd9f433f98cfef7169cdbe0c [id=2ff95cefc4a545398f979e2bd0321f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.715920 14722 tablet_bootstrap.cc:654] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.718963 14714 tablet_bootstrap.cc:492] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:37.704205 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 4cbd18e927c843e7b0aee7e0ccb1b06d (DEFAULT_TABLE table=default.Table_cb9f538930f64a52b0f18805535a8fc1 [id=a9e8baf63d354fdaab9fb2a0381ffbcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.719653 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cbd18e927c843e7b0aee7e0ccb1b06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.723999 14714 tablet_bootstrap.cc:654] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:37.724421 14422 tablet_service.cc:1505] Processing CreateTablet for tablet c6d0fef50524476dbd47fc70e4bda319 (DEFAULT_TABLE table=default.Table_7919ce70cfc5400bb4f2515164c6b7a1 [id=3a336d5ad7004619951e901ad9bf39bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.725353 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d0fef50524476dbd47fc70e4bda319. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.727360 14426 tablet_service.cc:1505] Processing CreateTablet for tablet da57b72207414512b9744b283bc869fe (DEFAULT_TABLE table=default.Table_2f325f51dd9f433f98cfef7169cdbe0c [id=2ff95cefc4a545398f979e2bd0321f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.728181 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da57b72207414512b9744b283bc869fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.731298 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 49015c5d4f2b4d07b547a9df57c992eb (DEFAULT_TABLE table=default.Table_73091d56834e4e78b870257988c87c57 [id=d8f07591554543f2880445cb636da15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.732077 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49015c5d4f2b4d07b547a9df57c992eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.737841 14008 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 3303dc521d694be6b386a35da1172ede (table default.Table_71b84842b7c84927ad89c5ad951be1a1 [id=ebfd6e4105a8410bb79a6db704ec0ba5]). Expected version 2 got 0
I20251211 08:57:37.740098 14268 tablet_service.cc:1505] Processing CreateTablet for tablet 49015c5d4f2b4d07b547a9df57c992eb (DEFAULT_TABLE table=default.Table_73091d56834e4e78b870257988c87c57 [id=d8f07591554543f2880445cb636da15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.741384 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 5f7775994f684326b6312c802f6b084d (DEFAULT_TABLE table=default.Table_c2b453de9d7c46ed92713e510df27dca [id=e5e62d49ddb749a29892b61cd4615da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.742377 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cbd18e927c843e7b0aee7e0ccb1b06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.744308 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 5f7775994f684326b6312c802f6b084d (DEFAULT_TABLE table=default.Table_c2b453de9d7c46ed92713e510df27dca [id=e5e62d49ddb749a29892b61cd4615da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.745234 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7775994f684326b6312c802f6b084d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.755029 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7775994f684326b6312c802f6b084d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.756431 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d0fef50524476dbd47fc70e4bda319. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.757782 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da57b72207414512b9744b283bc869fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.759572 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49015c5d4f2b4d07b547a9df57c992eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.761287 14271 tablet_service.cc:1505] Processing CreateTablet for tablet 98d14ba15dcd47f8b9e8035b3c7d477c (DEFAULT_TABLE table=default.Table_a4bc9b5633444f8587bf740793fd55a2 [id=1a961233d708456bb4cf8d3fc39dc705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.762159 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98d14ba15dcd47f8b9e8035b3c7d477c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.794220 14904 ts_tablet_manager.cc:1929] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:37.794534 14904 log.cc:1199] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/996c85729acd4418be1763b73cbf7d2d
I20251211 08:57:37.797048 14008 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet eb5a6e533e1f4da7b03a800d029f712d (table default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85]). Expected version 1 got 0
I20251211 08:57:37.798017 14722 tablet_bootstrap.cc:492] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:37.798453 14722 ts_tablet_manager.cc:1403] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.134s user 0.009s sys 0.000s
I20251211 08:57:37.798297 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 98d14ba15dcd47f8b9e8035b3c7d477c (DEFAULT_TABLE table=default.Table_a4bc9b5633444f8587bf740793fd55a2 [id=1a961233d708456bb4cf8d3fc39dc705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.799465 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98d14ba15dcd47f8b9e8035b3c7d477c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.808403 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 4b9ae463731d4828afc02039e0d161b5 (DEFAULT_TABLE table=default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.809660 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b9ae463731d4828afc02039e0d161b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.674722 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 150) took 2353 ms. Trace:
I20251211 08:57:37.796849 15275 raft_consensus.cc:493] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.819218 14905 rpcz_store.cc:276] 1211 08:57:35.321389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.322061 (+ 672us) service_pool.cc:225] Handling call
1211 08:57:37.553897 (+2231836us) ts_tablet_manager.cc:1151] Deleting tablet b8d753358f20476ba23d7d7b72e26e74
1211 08:57:37.553934 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.637180 (+ 83246us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:37.671714 (+ 34534us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:37.674673 (+ 2959us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2231299}
I20251211 08:57:37.819434 15275 raft_consensus.cc:515] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.820586 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 152) took 2371 ms. Trace:
I20251211 08:57:37.742790 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 5f7775994f684326b6312c802f6b084d (DEFAULT_TABLE table=default.Table_c2b453de9d7c46ed92713e510df27dca [id=e5e62d49ddb749a29892b61cd4615da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.820870 14905 rpcz_store.cc:276] 1211 08:57:35.448710 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.448819 (+ 109us) service_pool.cc:225] Handling call
1211 08:57:37.820277 (+2371458us) ts_tablet_manager.cc:1151] Deleting tablet db302282bc3d4d8cb56d1cbb2f73c4fe
1211 08:57:37.820318 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.820542 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2312822}
I20251211 08:57:37.821979 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 159) took 2075 ms. Trace:
I20251211 08:57:37.715981 14577 tablet_service.cc:1505] Processing CreateTablet for tablet da57b72207414512b9744b283bc869fe (DEFAULT_TABLE table=default.Table_2f325f51dd9f433f98cfef7169cdbe0c [id=2ff95cefc4a545398f979e2bd0321f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.822280 14905 rpcz_store.cc:276] 1211 08:57:35.746564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.750866 (+ 4302us) service_pool.cc:225] Handling call
1211 08:57:37.821723 (+2070857us) ts_tablet_manager.cc:1151] Deleting tablet c53035e83710403bac3a2b401c8c97ad
1211 08:57:37.821755 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.821936 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2070318}
W20251211 08:57:37.822708 14542 leader_election.cc:343] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 3c5315f708c04322919c714dc915a483
I20251211 08:57:37.679634 15030 tablet_bootstrap.cc:492] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:37.821986 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7775994f684326b6312c802f6b084d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.823483 15030 ts_tablet_manager.cc:1403] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.207s user 0.000s sys 0.014s
I20251211 08:57:37.824930 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c5315f708c04322919c714dc915a483" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:37.715833 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 49015c5d4f2b4d07b547a9df57c992eb (DEFAULT_TABLE table=default.Table_73091d56834e4e78b870257988c87c57 [id=d8f07591554543f2880445cb636da15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.825776 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49015c5d4f2b4d07b547a9df57c992eb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:37.826181 14544 leader_election.cc:343] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.826546 14544 leader_election.cc:304] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:37.772339 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet e6c4d4c99b3b496cab125e13153349cb
I20251211 08:57:37.827242 15237 raft_consensus.cc:2749] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:37.827831 15275 leader_election.cc:290] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:37.828092 14004 catalog_manager.cc:6724] ReplaceTablet: tablet e6c4d4c99b3b496cab125e13153349cb deleted and replaced by tablet 5d8d84e594ea47f3bc9127cc587e8aa7
I20251211 08:57:37.828195 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet e6c4d4c99b3b496cab125e13153349cb with delete_type TABLET_DATA_DELETED (replaced by tablet 5d8d84e594ea47f3bc9127cc587e8aa7) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:37.829057 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_03bee763242749cfa5fb0fe7d7e7fb7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:37.830451 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03bee763242749cfa5fb0fe7d7e7fb7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:37.702922 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 4cbd18e927c843e7b0aee7e0ccb1b06d (DEFAULT_TABLE table=default.Table_cb9f538930f64a52b0f18805535a8fc1 [id=a9e8baf63d354fdaab9fb2a0381ffbcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.831737 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cbd18e927c843e7b0aee7e0ccb1b06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.832547 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet e6c4d4c99b3b496cab125e13153349cb with delete_type TABLET_DATA_DELETED (replaced by tablet 5d8d84e594ea47f3bc9127cc587e8aa7) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:37.744123 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet afe339c4cc95490683618e18a3da252b
I20251211 08:57:37.820613 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet b8d753358f20476ba23d7d7b72e26e74 (table default.Table_6cbe43becdc846bcab4b1233f65cda0c [id=8a6e51e5adbf4ca39770244bfdc843bc]) successfully deleted
I20251211 08:57:37.833658 14722 raft_consensus.cc:359] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.834160 14722 raft_consensus.cc:385] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.834357 14722 raft_consensus.cc:740] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
W20251211 08:57:37.834501 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet db302282bc3d4d8cb56d1cbb2f73c4fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db302282bc3d4d8cb56d1cbb2f73c4fe already in progress: creating tablet
I20251211 08:57:37.686465 15152 raft_consensus.cc:493] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.834888 14722 consensus_queue.cc:260] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.835309 14579 tablet_service.cc:1552] Processing DeleteTablet for tablet e6c4d4c99b3b496cab125e13153349cb with delete_type TABLET_DATA_DELETED (replaced by tablet 5d8d84e594ea47f3bc9127cc587e8aa7) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:37.711866 14576 tablet_service.cc:1505] Processing CreateTablet for tablet c6d0fef50524476dbd47fc70e4bda319 (DEFAULT_TABLE table=default.Table_7919ce70cfc5400bb4f2515164c6b7a1 [id=3a336d5ad7004619951e901ad9bf39bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.836124 14007 catalog_manager.cc:6724] ReplaceTablet: tablet afe339c4cc95490683618e18a3da252b deleted and replaced by tablet e36c6e2868e143be8e7f72307202a268
I20251211 08:57:37.836869 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d0fef50524476dbd47fc70e4bda319. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.836812 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_05c727e9a723434ba4239e838c6afc9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:37.838047 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05c727e9a723434ba4239e838c6afc9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:37.838816 15193 raft_consensus.cc:493] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.839234 15193 raft_consensus.cc:515] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:37.839648 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet c53035e83710403bac3a2b401c8c97ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c53035e83710403bac3a2b401c8c97ad already in progress: creating tablet
I20251211 08:57:37.840780 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet afe339c4cc95490683618e18a3da252b with delete_type TABLET_DATA_DELETED (replaced by tablet e36c6e2868e143be8e7f72307202a268) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:37.840919 14579 tablet_service.cc:1552] Processing DeleteTablet for tablet afe339c4cc95490683618e18a3da252b with delete_type TABLET_DATA_DELETED (replaced by tablet e36c6e2868e143be8e7f72307202a268) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:37.841130 15193 leader_election.cc:290] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:37.842412 14417 tablet_service.cc:1552] Processing DeleteTablet for tablet afe339c4cc95490683618e18a3da252b with delete_type TABLET_DATA_DELETED (replaced by tablet e36c6e2868e143be8e7f72307202a268) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:37.842455 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42720f5f5bf445f2b72e513ad4a6d95f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:37.708829 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 45c7d841bd3643a799a8b96e82d6d675 (DEFAULT_TABLE table=default.Table_4960eb1f338e47b3923b57ce8be73703 [id=f08b1a30da03451ba864cb8f8e6d199b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.843041 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42720f5f5bf445f2b72e513ad4a6d95f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:37.804912 14581 tablet_service.cc:1505] Processing CreateTablet for tablet 98d14ba15dcd47f8b9e8035b3c7d477c (DEFAULT_TABLE table=default.Table_a4bc9b5633444f8587bf740793fd55a2 [id=1a961233d708456bb4cf8d3fc39dc705]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:37.843636 14382 leader_election.cc:343] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.843446 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45c7d841bd3643a799a8b96e82d6d675. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:37.844130 14385 leader_election.cc:343] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.844441 14385 leader_election.cc:304] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:37.844794 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98d14ba15dcd47f8b9e8035b3c7d477c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.845422 15193 raft_consensus.cc:2749] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:37.847242 14904 ts_tablet_manager.cc:1950] T 996c85729acd4418be1763b73cbf7d2d P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:37.848161 14417 tablet_service.cc:1505] Processing CreateTablet for tablet 4b9ae463731d4828afc02039e0d161b5 (DEFAULT_TABLE table=default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.848898 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet c341b762afd8440bb7198ca134c8dffe
I20251211 08:57:37.849608 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da57b72207414512b9744b283bc869fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.850852 14579 tablet_service.cc:1505] Processing CreateTablet for tablet 4b9ae463731d4828afc02039e0d161b5 (DEFAULT_TABLE table=default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.850499 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_fd201f899f064ec0b339bd8b7679fefc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:37.852223 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b9ae463731d4828afc02039e0d161b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.852562 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet c341b762afd8440bb7198ca134c8dffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:37 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:37.852743 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd201f899f064ec0b339bd8b7679fefc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:37.835100 15152 raft_consensus.cc:515] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.854158 14582 tablet_service.cc:1552] Processing DeleteTablet for tablet c341b762afd8440bb7198ca134c8dffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:37 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:37.854290 14008 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 6de43c843ff845a0bd4eb98b6054dd87 (table default.Table_3c7e457251e74eb7b8415bbcb6b6d7db [id=e5ffdeadfa904d85b68d0925da7e7a4c]). Expected version 1 got 0
I20251211 08:57:37.855134 15152 leader_election.cc:290] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:37.855839 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "363fe7a2e1ac4fb4a1ec97dc2354d71a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:37.856371 14446 raft_consensus.cc:2468] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:37.858084 14542 leader_election.cc:304] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:37.860157 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "363fe7a2e1ac4fb4a1ec97dc2354d71a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:37.861264 14544 leader_election.cc:343] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.860677 15152 raft_consensus.cc:2804] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:37.861959 15152 raft_consensus.cc:493] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:37.862140 14417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b9ae463731d4828afc02039e0d161b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.862303 15152 raft_consensus.cc:3060] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:37.862960 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet c341b762afd8440bb7198ca134c8dffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:37 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:37.872484 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 157) took 2138 ms. Trace:
I20251211 08:57:37.872769 14271 tablet_service.cc:1505] Processing CreateTablet for tablet 4ab48f6288494005ad42208e2e97ddf2 (DEFAULT_TABLE table=default.Table_ea1935de13854b0b82e1382effe6cfbb [id=658ce49773af41519ab660f3bc29975b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.873696 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab48f6288494005ad42208e2e97ddf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.881103 15152 raft_consensus.cc:515] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.881872 15228 raft_consensus.cc:493] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.882467 15030 raft_consensus.cc:359] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.882340 15228 raft_consensus.cc:515] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.883071 15030 raft_consensus.cc:385] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:37.883303 15030 raft_consensus.cc:740] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:37.883919 15030 consensus_queue.cc:260] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.884825 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afe339c4cc95490683618e18a3da252b" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:37.885735 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "363fe7a2e1ac4fb4a1ec97dc2354d71a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:37.886233 14446 raft_consensus.cc:3060] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:37.894105 15030 ts_tablet_manager.cc:1434] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.070s user 0.004s sys 0.001s
I20251211 08:57:37.872787 14905 rpcz_store.cc:276] 1211 08:57:35.733978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.844029 (+110051us) service_pool.cc:225] Handling call
1211 08:57:37.823119 (+1979090us) ts_tablet_manager.cc:1151] Deleting tablet 42720f5f5bf445f2b72e513ad4a6d95f
1211 08:57:37.872152 (+ 49033us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.872435 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1960171}
W20251211 08:57:37.903671 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 42720f5f5bf445f2b72e513ad4a6d95f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42720f5f5bf445f2b72e513ad4a6d95f already in progress: creating tablet
I20251211 08:57:37.904479 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 166) took 1999 ms. Trace:
I20251211 08:57:37.904623 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 383716a1f1f847ccbea542eca04790c9 (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]). Expected version 2 got 0
I20251211 08:57:37.904716 14905 rpcz_store.cc:276] 1211 08:57:35.905154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.910929 (+ 5775us) service_pool.cc:225] Handling call
1211 08:57:37.904194 (+1993265us) ts_tablet_manager.cc:1151] Deleting tablet 174fefd70bcc41418a77a5716f97cd5f
1211 08:57:37.904230 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.904437 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1992211}
I20251211 08:57:37.905458 14905 tablet_replica.cc:333] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:37.905736 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 3303dc521d694be6b386a35da1172ede (table default.Table_71b84842b7c84927ad89c5ad951be1a1 [id=ebfd6e4105a8410bb79a6db704ec0ba5]). Expected version 2 got 0
W20251211 08:57:37.906026 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 174fefd70bcc41418a77a5716f97cd5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 174fefd70bcc41418a77a5716f97cd5f already in progress: creating tablet
I20251211 08:57:37.911139 14009 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 3303dc521d694be6b386a35da1172ede (table default.Table_71b84842b7c84927ad89c5ad951be1a1 [id=ebfd6e4105a8410bb79a6db704ec0ba5]). Expected version 2 got 0
I20251211 08:57:37.924028 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7d2f7854e92d464cbebec8b6406ab053" } new_table_name: "default.Table_eb3b1d93a3ec4891a9d33b131661fd65" modify_external_catalogs: true
I20251211 08:57:37.926062 15030 tablet_bootstrap.cc:492] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:37.934360 14446 raft_consensus.cc:2468] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:37.935475 14542 leader_election.cc:304] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:37.936395 15275 raft_consensus.cc:2804] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:37.937057 15275 raft_consensus.cc:697] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:37.937969 15275 consensus_queue.cc:237] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.944644 14722 ts_tablet_manager.cc:1434] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.146s user 0.006s sys 0.000s
I20251211 08:57:37.946235 14722 tablet_bootstrap.cc:492] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:37.906621 14905 raft_consensus.cc:2243] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:37.947131 14905 raft_consensus.cc:2272] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:37.948776 14905 ts_tablet_manager.cc:1916] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:37.932950 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_10e56e6be8ad4d798a3d979bbb66ffc3" } new_table_name: "default.Table_e66c701250064c388e221a300dac3dd0" modify_external_catalogs: true
I20251211 08:57:37.925459 14013 catalog_manager.cc:5654] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:37.953101 15193 consensus_queue.cc:1048] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:37.954242 14418 tablet_service.cc:1505] Processing CreateTablet for tablet 4ab48f6288494005ad42208e2e97ddf2 (DEFAULT_TABLE table=default.Table_ea1935de13854b0b82e1382effe6cfbb [id=658ce49773af41519ab660f3bc29975b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.954372 14013 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 40a1f12553934446bfe44f4ed41abf9a (table default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127]). Expected version 1 got 0
I20251211 08:57:37.956389 14582 tablet_service.cc:1505] Processing CreateTablet for tablet 4ab48f6288494005ad42208e2e97ddf2 (DEFAULT_TABLE table=default.Table_ea1935de13854b0b82e1382effe6cfbb [id=658ce49773af41519ab660f3bc29975b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:37.957357 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab48f6288494005ad42208e2e97ddf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:37.933574 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_02db0ed6d7e34eb388c5e24bdf50bf74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:37.959743 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02db0ed6d7e34eb388c5e24bdf50bf74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:37.969489 15193 raft_consensus.cc:493] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.969910 15193 raft_consensus.cc:515] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.980101 15193 leader_election.cc:290] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:37.980162 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f826d079774152a3b66bfb81f990a8" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:37.981344 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f826d079774152a3b66bfb81f990a8" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:37.982904 15148 raft_consensus.cc:493] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:37.983358 15148 raft_consensus.cc:515] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:37.987490 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf537c461c04a81b66f268ee98dc64e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:37.987589 14382 leader_election.cc:343] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:37.989058 14385 leader_election.cc:343] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:37.989424 14385 leader_election.cc:304] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:37.989831 14446 raft_consensus.cc:2468] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:37.991091 14231 leader_election.cc:304] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:38.001015 15030 tablet_bootstrap.cc:654] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:38.000948 15193 raft_consensus.cc:2749] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:38.003909 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab48f6288494005ad42208e2e97ddf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.010244 14266 tablet_service.cc:1505] Processing CreateTablet for tablet dd14de369e084828afb8a924c46c649b (DEFAULT_TABLE table=default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:38.021153 14004 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed0e90efe651499c929b1fb992712b96 has no consensus state
W20251211 08:57:38.022298 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 578404732eef4e51b66b6aebffa87cdb consensus state has no leader
I20251211 08:57:38.028478 15299 tablet_replica.cc:333] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:38.029418 15299 raft_consensus.cc:2243] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:38.029975 15299 raft_consensus.cc:2272] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:38.031991 15299 ts_tablet_manager.cc:1916] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:37.955868 14013 catalog_manager.cc:5654] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:38.033419 14013 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 383716a1f1f847ccbea542eca04790c9 (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]). Expected version 2 got 0
I20251211 08:57:38.034173 14010 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 81cfb3e0baef45ddab4a77a4e0790085
I20251211 08:57:38.035531 14010 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55614 (request call id 237) took 1592 ms. Trace:
I20251211 08:57:38.035833 14010 rpcz_store.cc:276] 1211 08:57:36.443556 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.443661 (+ 105us) service_pool.cc:225] Handling call
1211 08:57:36.451645 (+ 7984us) spinlock_profiling.cc:243] Waited 7.31 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da86ca6 00007fc20b6b1024 00007fc20b6b0fb2 00007fc20b6b0ee1 00007fc20b6b0e5c 00007fc20b6b0112 00007fc20998455c 00007fc209983996
1211 08:57:36.452545 (+ 900us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:36.453017 (+ 472us) catalog_manager.cc:2815] Modifying in-memory table state
1211 08:57:36.453129 (+ 112us) catalog_manager.cc:2824] Locking tablets
1211 08:57:36.453674 (+ 545us) catalog_manager.cc:2838] Removing table and tablets from system table
1211 08:57:36.888406 (+434732us) catalog_manager.cc:2856] Removing table from by-name map
1211 08:57:36.888621 (+ 215us) catalog_manager.cc:2872] Committing in-memory state
1211 08:57:38.034096 (+1145475us) catalog_manager.cc:2876] Aborting table tasks
1211 08:57:38.035479 (+ 1383us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:36.842176 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:36.842853 (+ 677us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:36.842879 (+ 26us) write_op.cc:435] Acquired schema lock
1211 08:57:36.842889 (+ 10us) tablet.cc:660] Decoding operations
1211 08:57:36.843139 (+ 250us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:36.843167 (+ 28us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:36.843185 (+ 18us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:36.843376 (+ 191us) tablet.cc:699] Row locks acquired
1211 08:57:36.843388 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:57:36.843475 (+ 87us) write_op.cc:270] Start()
1211 08:57:36.843537 (+ 62us) write_op.cc:276] Timestamp: P: 1765443456843537 usec, L: 0
1211 08:57:36.843549 (+ 12us) op_driver.cc:348] REPLICATION: starting
1211 08:57:36.843911 (+ 362us) log.cc:844] Serialized 728 byte log entry
1211 08:57:36.884288 (+ 40377us) op_driver.cc:464] REPLICATION: finished
1211 08:57:36.886246 (+ 1958us) write_op.cc:301] APPLY: starting
1211 08:57:36.886288 (+ 42us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:36.886391 (+ 103us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:36.886476 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:36.886678 (+ 202us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:36.886835 (+ 157us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:36.886897 (+ 62us) write_op.cc:312] APPLY: finished
1211 08:57:36.887099 (+ 202us) log.cc:844] Serialized 29 byte log entry
1211 08:57:36.887252 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:36.887330 (+ 78us) write_op.cc:454] Released schema lock
1211 08:57:36.887423 (+ 93us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":16820992,"child_traces":[["op",{"apply.queue_time_us":1846,"apply.run_cpu_time_us":1373,"apply.run_wall_time_us":2522,"num_ops":2,"prepare.queue_time_us":380063,"prepare.run_cpu_time_us":1580,"prepare.run_wall_time_us":3196,"replication_time_us":40665,"thread_start_us":24818,"threads_started":1}]]}
I20251211 08:57:38.054672 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_0551d84bbd1b41399aab17ec5d23007c" } modify_external_catalogs: true
I20251211 08:57:38.055282 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_0551d84bbd1b41399aab17ec5d23007c" } modify_external_catalogs: true
I20251211 08:57:38.060999 14585 tablet_service.cc:1505] Processing CreateTablet for tablet dd14de369e084828afb8a924c46c649b (DEFAULT_TABLE table=default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.062072 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd14de369e084828afb8a924c46c649b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.064930 15152 leader_election.cc:290] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:38.066115 14579 tablet_service.cc:1552] Processing DeleteTablet for tablet 81cfb3e0baef45ddab4a77a4e0790085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:38.067220 14009 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_3c7e457251e74eb7b8415bbcb6b6d7db [id=e5ffdeadfa904d85b68d0925da7e7a4c] alter complete (version 1)
I20251211 08:57:38.073072 14012 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_2d7127602bd94e2dbc13d3cfae809a87 [id=e9973d6c9b1341d1b96f04367e44e212] alter complete (version 2)
W20251211 08:57:38.073619 14012 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 30) took 1877 ms (client timeout 1000 ms). Trace:
W20251211 08:57:38.073895 14012 rpcz_store.cc:269] 1211 08:57:36.196250 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.196352 (+ 102us) service_pool.cc:225] Handling call
1211 08:57:36.305384 (+109032us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:38.073562 (+1768178us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:36.484603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:36.484974 (+ 371us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:36.485005 (+ 31us) write_op.cc:435] Acquired schema lock
1211 08:57:36.485025 (+ 20us) tablet.cc:660] Decoding operations
1211 08:57:36.485328 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:36.485380 (+ 52us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:36.485413 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:36.485579 (+ 166us) tablet.cc:699] Row locks acquired
1211 08:57:36.485598 (+ 19us) write_op.cc:260] PREPARE: finished
1211 08:57:36.485738 (+ 140us) write_op.cc:270] Start()
1211 08:57:36.485836 (+ 98us) write_op.cc:276] Timestamp: P: 1765443456485841 usec, L: 0
1211 08:57:36.485859 (+ 23us) op_driver.cc:348] REPLICATION: starting
1211 08:57:36.486501 (+ 642us) log.cc:844] Serialized 356 byte log entry
1211 08:57:36.541676 (+ 55175us) op_driver.cc:464] REPLICATION: finished
1211 08:57:36.542378 (+ 702us) write_op.cc:301] APPLY: starting
1211 08:57:36.542431 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:36.542510 (+ 79us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:36.542583 (+ 73us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:36.542728 (+ 145us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:36.542921 (+ 193us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:36.542958 (+ 37us) write_op.cc:312] APPLY: finished
1211 08:57:36.543099 (+ 141us) log.cc:844] Serialized 23 byte log entry
1211 08:57:36.543634 (+ 535us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:36.543699 (+ 65us) write_op.cc:454] Released schema lock
1211 08:57:36.543725 (+ 26us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:36.554750 (+ 11025us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:36.554770 (+ 20us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":247037824,"child_traces":[["op",{"apply.queue_time_us":559,"apply.run_cpu_time_us":1066,"apply.run_wall_time_us":1613,"num_ops":1,"prepare.queue_time_us":102888,"prepare.run_cpu_time_us":2200,"prepare.run_wall_time_us":70142,"raft.queue_time_us":66,"raft.run_cpu_time_us":604,"raft.run_wall_time_us":1451,"replication_time_us":55714}]]}
W20251211 08:57:38.080044 14011 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 38) took 1872 ms (client timeout 1000 ms). Trace:
W20251211 08:57:38.080348 14011 rpcz_store.cc:269] 1211 08:57:36.207587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.207700 (+ 113us) service_pool.cc:225] Handling call
1211 08:57:36.303319 (+ 95619us) spinlock_profiling.cc:243] Waited 93.9 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:38.079987 (+1776668us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6695,"reported_tablets":4,"spinlock_wait_cycles":215915392}
I20251211 08:57:38.087152 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 4eec7694087145d8a49dd93904f95980
I20251211 08:57:38.088039 14580 tablet_service.cc:1505] Processing CreateTablet for tablet da52cace2aab4f9dacef18ec5fa56aaf (DEFAULT_TABLE table=default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.089241 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da52cace2aab4f9dacef18ec5fa56aaf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:38.090657 13999 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 -> Peer 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015): Couldn't send request to peer 73702d54989f4959acfc4a6fdcede6ca. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20251211 08:57:38.096333 14009 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85] alter complete (version 1)
I20251211 08:57:38.103875 14579 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eec7694087145d8a49dd93904f95980 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:38.107584 15030 tablet_bootstrap.cc:492] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:38.108052 15030 ts_tablet_manager.cc:1403] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.213s user 0.012s sys 0.000s
I20251211 08:57:38.111254 15030 raft_consensus.cc:359] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.111963 15030 raft_consensus.cc:385] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:38.112244 15030 raft_consensus.cc:740] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:38.045418 15083 raft_consensus.cc:493] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:38.112736 15083 raft_consensus.cc:515] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.112890 15030 consensus_queue.cc:260] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.046274 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 81cfb3e0baef45ddab4a77a4e0790085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:38.062088 14419 tablet_service.cc:1505] Processing CreateTablet for tablet da52cace2aab4f9dacef18ec5fa56aaf (DEFAULT_TABLE table=default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.066452 14428 tablet_service.cc:1505] Processing CreateTablet for tablet dd14de369e084828afb8a924c46c649b (DEFAULT_TABLE table=default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.108011 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 1976d584cc0a450ab809ead258a251e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:38.114475 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da52cace2aab4f9dacef18ec5fa56aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.114955 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd14de369e084828afb8a924c46c649b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.127929 15083 leader_election.cc:290] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:38.134166 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a1f12553934446bfe44f4ed41abf9a" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:38.134852 14607 raft_consensus.cc:2468] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:38.135746 14382 leader_election.cc:304] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:38.136646 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_80758110ce794260af2a551849bf4896"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:38.138146 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80758110ce794260af2a551849bf4896 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:38.141690 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7343d67071c54141af5a57615f82221e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:38.142071 14579 tablet_service.cc:1505] Processing CreateTablet for tablet 080d1700df4a4df09acda65c6d021d96 (DEFAULT_TABLE table=default.Table_26f4e336df6947149cb311c15bee56c1 [id=e647fb169f024aebb1f942fff0779130]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.143024 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_417cd1cc5e354dddaa9d5c0c3f8ddd8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:38.143399 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080d1700df4a4df09acda65c6d021d96. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:38.143294 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7343d67071c54141af5a57615f82221e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:38.152892 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 162033us
I20251211 08:57:38.153116 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 81cfb3e0baef45ddab4a77a4e0790085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:38.153636 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "363fe7a2e1ac4fb4a1ec97dc2354d71a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:38.159502 14005 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_da9fb81b38a14423b6191db9f7a7c3df [id=e10eaa167f0a46c39674a5e2212f2226] alter complete (version 1)
W20251211 08:57:38.160573 14230 leader_election.cc:343] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: caf537c461c04a81b66f268ee98dc64e
W20251211 08:57:38.163793 14009 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 39) took 883 ms (client timeout 1000 ms). Trace:
I20251211 08:57:38.165597 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 552de4c6d030418181f0400714d9ec31 (DEFAULT_TABLE table=default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.166543 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 552de4c6d030418181f0400714d9ec31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.167095 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afe339c4cc95490683618e18a3da252b" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:38.168267 14230 leader_election.cc:304] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:38.167893 14582 tablet_service.cc:1505] Processing CreateTablet for tablet cb2ff90b54544420b1aad9d2ab4132ff (DEFAULT_TABLE table=default.Table_25fc063a532541f5a7d6c73d8f2f0e33 [id=3c0379138b1c48e0a5400a465c4f372e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.169183 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:31 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:38.169577 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb2ff90b54544420b1aad9d2ab4132ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.169854 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 174) took 1177 ms. Trace:
I20251211 08:57:38.171211 14274 tablet_service.cc:1505] Processing CreateTablet for tablet da52cace2aab4f9dacef18ec5fa56aaf (DEFAULT_TABLE table=default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:38.164158 14009 rpcz_store.cc:269] 1211 08:57:37.280362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:37.294842 (+ 14480us) service_pool.cc:225] Handling call
1211 08:57:37.309809 (+ 14967us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:38.163740 (+853931us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:37.941135 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:37.941547 (+ 412us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:37.941570 (+ 23us) write_op.cc:435] Acquired schema lock
1211 08:57:37.941583 (+ 13us) tablet.cc:660] Decoding operations
1211 08:57:37.941811 (+ 228us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:37.941849 (+ 38us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:37.941871 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:37.942000 (+ 129us) tablet.cc:699] Row locks acquired
1211 08:57:37.942014 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:57:37.942110 (+ 96us) write_op.cc:270] Start()
1211 08:57:37.942187 (+ 77us) write_op.cc:276] Timestamp: P: 1765443457942195 usec, L: 0
1211 08:57:37.942209 (+ 22us) op_driver.cc:348] REPLICATION: starting
1211 08:57:37.942687 (+ 478us) log.cc:844] Serialized 356 byte log entry
1211 08:57:37.951204 (+ 8517us) op_driver.cc:464] REPLICATION: finished
1211 08:57:37.951443 (+ 239us) write_op.cc:301] APPLY: starting
1211 08:57:37.951494 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:37.951573 (+ 79us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:37.951641 (+ 68us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:37.951770 (+ 129us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:37.951880 (+ 110us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:37.951930 (+ 50us) write_op.cc:312] APPLY: finished
1211 08:57:37.952059 (+ 129us) log.cc:844] Serialized 23 byte log entry
1211 08:57:37.952787 (+ 728us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:37.952838 (+ 51us) write_op.cc:454] Released schema lock
1211 08:57:37.952863 (+ 25us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:37.953049 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:37.953068 (+ 19us) write_op.cc:454] Released schema lock
Related trace 'op':
1211 08:57:38.085036 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:38.085329 (+ 293us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:38.085349 (+ 20us) write_op.cc:435] Acquired schema lock
1211 08:57:38.085361 (+ 12us) tablet.cc:660] Decoding operations
1211 08:57:38.085582 (+ 221us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:38.085615 (+ 33us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:38.085635 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:38.085749 (+ 114us) tablet.cc:699] Row locks acquired
1211 08:57:38.085772 (+ 23us) write_op.cc:260] PREPARE: finished
1211 08:57:38.085858 (+ 86us) write_op.cc:270] Start()
1211 08:57:38.085931 (+ 73us) write_op.cc:276] Timestamp: P: 1765443458086005 usec, L: 0
1211 08:57:38.085946 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:57:38.086313 (+ 367us) log.cc:844] Serialized 356 byte log entry
1211 08:57:38.094213 (+ 7900us) op_driver.cc:464] REPLICATION: finished
1211 08:57:38.094571 (+ 358us) write_op.cc:301] APPLY: starting
1211 08:57:38.094617 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:38.094698 (+ 81us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:38.094823 (+ 125us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:38.094984 (+ 161us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:38.095107 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:38.095150 (+ 43us) write_op.cc:312] APPLY: finished
1211 08:57:38.095326 (+ 176us) log.cc:844] Serialized 23 byte log entry
1211 08:57:38.095455 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:38.095515 (+ 60us) write_op.cc:454] Released schema lock
1211 08:57:38.095545 (+ 30us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:38.095756 (+ 211us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:38.095781 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7188,"reported_tablets":10,"spinlock_wait_cycles":28139776,"child_traces":[["op",{"apply.queue_time_us":72,"apply.run_cpu_time_us":974,"apply.run_wall_time_us":1578,"num_ops":1,"prepare.queue_time_us":21395,"prepare.run_cpu_time_us":1830,"prepare.run_wall_time_us":1829,"raft.queue_time_us":167,"raft.run_cpu_time_us":601,"raft.run_wall_time_us":1639,"replication_time_us":8898}],["op",{"apply.queue_time_us":226,"apply.run_cpu_time_us":1150,"apply.run_wall_time_us":1149,"num_ops":1,"prepare.queue_time_us":13637,"prepare.run_cpu_time_us":1530,"prepare.run_wall_time_us":3676,"raft.queue_time_us":1551,"raft.run_cpu_time_us":494,"raft.run_wall_time_us":494,"replication_time_us":8171}]]}
I20251211 08:57:38.156351 15148 leader_election.cc:290] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:38.174656 14008 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 28) took 1657 ms (client timeout 1000 ms). Trace:
W20251211 08:57:38.146678 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_417cd1cc5e354dddaa9d5c0c3f8ddd8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:38.174994 14008 rpcz_store.cc:269] 1211 08:57:36.517578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.517672 (+ 94us) service_pool.cc:225] Handling call
1211 08:57:38.174607 (+1656935us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":11,"spinlock_wait_cycles":1920}
I20251211 08:57:38.170154 14904 rpcz_store.cc:276] 1211 08:57:36.992776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.993463 (+ 687us) service_pool.cc:225] Handling call
1211 08:57:37.604681 (+611218us) ts_tablet_manager.cc:1151] Deleting tablet 996c85729acd4418be1763b73cbf7d2d
1211 08:57:37.604728 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:37.682226 (+ 77498us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:37.794024 (+111798us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:38.169802 (+375778us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":90210,"tablet-delete.queue_time_us":580330}
I20251211 08:57:38.177572 14904 tablet_replica.cc:333] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:38.178268 14904 raft_consensus.cc:2243] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:38.178658 14904 raft_consensus.cc:2272] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:38.180176 14904 ts_tablet_manager.cc:1916] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:38.166978 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd14de369e084828afb8a924c46c649b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.180352 15030 ts_tablet_manager.cc:1434] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.072s user 0.005s sys 0.000s
I20251211 08:57:38.181216 15030 tablet_bootstrap.cc:492] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:38.165643 15228 leader_election.cc:290] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:38.182359 14905 ts_tablet_manager.cc:1929] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:38.182343 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eec7694087145d8a49dd93904f95980 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:38.182680 14905 log.cc:1199] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/479d7fe06a284234aef420e3842f0eab
I20251211 08:57:38.182355 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 080d1700df4a4df09acda65c6d021d96 (DEFAULT_TABLE table=default.Table_26f4e336df6947149cb311c15bee56c1 [id=e647fb169f024aebb1f942fff0779130]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.183346 14905 ts_tablet_manager.cc:1950] T 479d7fe06a284234aef420e3842f0eab P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:38.184573 14277 tablet_service.cc:1505] Processing CreateTablet for tablet cb2ff90b54544420b1aad9d2ab4132ff (DEFAULT_TABLE table=default.Table_25fc063a532541f5a7d6c73d8f2f0e33 [id=3c0379138b1c48e0a5400a465c4f372e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.185304 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 552de4c6d030418181f0400714d9ec31 (DEFAULT_TABLE table=default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.186218 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 996c85729acd4418be1763b73cbf7d2d (table default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]) successfully deleted
I20251211 08:57:38.186995 14013 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_71b84842b7c84927ad89c5ad951be1a1 [id=ebfd6e4105a8410bb79a6db704ec0ba5] alter complete (version 2)
W20251211 08:57:38.193580 14005 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 31) took 986 ms (client timeout 1000 ms). Trace:
I20251211 08:57:38.208570 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb2ff90b54544420b1aad9d2ab4132ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.209777 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 552de4c6d030418181f0400714d9ec31. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:38.193970 14005 rpcz_store.cc:269] 1211 08:57:37.207105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:37.207246 (+ 141us) service_pool.cc:225] Handling call
1211 08:57:38.193518 (+986272us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:38.142057 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:38.144098 (+ 2041us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:38.144120 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:57:38.144143 (+ 23us) tablet.cc:660] Decoding operations
1211 08:57:38.144351 (+ 208us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:38.144384 (+ 33us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:38.144403 (+ 19us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:38.144516 (+ 113us) tablet.cc:699] Row locks acquired
1211 08:57:38.144527 (+ 11us) write_op.cc:260] PREPARE: finished
1211 08:57:38.144615 (+ 88us) write_op.cc:270] Start()
1211 08:57:38.144680 (+ 65us) write_op.cc:276] Timestamp: P: 1765443458144786 usec, L: 0
1211 08:57:38.144693 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:38.145067 (+ 374us) log.cc:844] Serialized 356 byte log entry
1211 08:57:38.157933 (+ 12866us) op_driver.cc:464] REPLICATION: finished
1211 08:57:38.158329 (+ 396us) write_op.cc:301] APPLY: starting
1211 08:57:38.158375 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:38.158462 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:38.158537 (+ 75us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:38.158687 (+ 150us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:38.158817 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:38.158856 (+ 39us) write_op.cc:312] APPLY: finished
1211 08:57:38.159017 (+ 161us) log.cc:844] Serialized 23 byte log entry
1211 08:57:38.159148 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:38.159205 (+ 57us) write_op.cc:454] Released schema lock
1211 08:57:38.159233 (+ 28us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:38.173074 (+ 13841us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:38.173099 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":278,"apply.run_cpu_time_us":1072,"apply.run_wall_time_us":1072,"num_ops":1,"prepare.queue_time_us":40010,"prepare.run_cpu_time_us":1674,"prepare.run_wall_time_us":30988,"raft.queue_time_us":3188,"raft.run_cpu_time_us":559,"raft.run_wall_time_us":1336,"replication_time_us":13149}]]}
I20251211 08:57:38.211184 14722 tablet_bootstrap.cc:654] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:38.221208 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da52cace2aab4f9dacef18ec5fa56aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.227167 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080d1700df4a4df09acda65c6d021d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.230178 14011 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_fa2830322a8a4f7b96cd19543b967b05" } new_table_name: "default.Table_c9d62497a342488290749d4f7c02e748" modify_external_catalogs: true
I20251211 08:57:38.241832 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f627382c623a4f818caf527706e776ab" } modify_external_catalogs: true
I20251211 08:57:38.242512 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f627382c623a4f818caf527706e776ab" } modify_external_catalogs: true
W20251211 08:57:38.243038 13999 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 98d14ba15dcd47f8b9e8035b3c7d477c (table default.Table_a4bc9b5633444f8587bf740793fd55a2 [id=1a961233d708456bb4cf8d3fc39dc705], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 98d14ba15dcd47f8b9e8035b3c7d477c consensus state has no leader
I20251211 08:57:38.256697 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_626342780a404385ad91b9e642a77ed8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:38.257982 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_626342780a404385ad91b9e642a77ed8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:38.265198 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4960eb1f338e47b3923b57ce8be73703" } new_table_name: "default.Table_f1453eb82ce2412c8208000d8ba7f240" modify_external_catalogs: true
I20251211 08:57:38.181772 15302 raft_consensus.cc:2804] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:38.214622 15148 raft_consensus.cc:2749] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:38.313627 14904 ts_tablet_manager.cc:1929] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:38.333487 14904 log.cc:1199] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/e6c4d4c99b3b496cab125e13153349cb
I20251211 08:57:38.334192 14904 ts_tablet_manager.cc:1950] T e6c4d4c99b3b496cab125e13153349cb P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:38.339066 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a1f12553934446bfe44f4ed41abf9a" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:38.339447 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 (DEFAULT_TABLE table=default.Table_d40cb8a6b4974e2e9ff982e893a2c552 [id=d6acd0e03f2743349b345927f5ce7a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.340489 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07aef1eea7c04d9cbf35c82e45df7a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.342003 14588 tablet_service.cc:1505] Processing CreateTablet for tablet e9905227c0a844d199362a442e7d630c (DEFAULT_TABLE table=default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b [id=45f67c67257e428f8ed2f16898ad5fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.342456 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 (DEFAULT_TABLE table=default.Table_d40cb8a6b4974e2e9ff982e893a2c552 [id=d6acd0e03f2743349b345927f5ce7a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.343653 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9905227c0a844d199362a442e7d630c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.343781 14576 tablet_service.cc:1505] Processing CreateTablet for tablet 17ab48721a5c48a7b8a5550be658837d (DEFAULT_TABLE table=default.Table_f13b896c35754c0a94cc5b238778ecd2 [id=1b741ee1e2bd404f982e2c2aea8da710]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.344626 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17ab48721a5c48a7b8a5550be658837d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.347421 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 8e05628011634149b17d298c54074d00 (DEFAULT_TABLE table=default.Table_885d3fb0ab5f4d0a82972d94a34c2a88 [id=18d1df6385a24d26bfb44309aa58eaa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.348392 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e05628011634149b17d298c54074d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.353349 14587 tablet_service.cc:1505] Processing CreateTablet for tablet c341b762afd8440bb7198ca134c8dffe (DEFAULT_TABLE table=default.Table_96d05b744df34836be68b4841e6c2a2b [id=86d8260b23ca4945acd9a70054b77c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.354152 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c341b762afd8440bb7198ca134c8dffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.350628 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eec7694087145d8a49dd93904f95980 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:38.349124 15083 raft_consensus.cc:2804] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:38.361188 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_885d3fb0ab5f4d0a82972d94a34c2a88" } modify_external_catalogs: true
I20251211 08:57:38.361626 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_885d3fb0ab5f4d0a82972d94a34c2a88" } modify_external_catalogs: true
W20251211 08:57:38.363091 14011 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41d6ef975a47489385791065bb8f9f46 consensus state has no leader
I20251211 08:57:38.367480 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 8e05628011634149b17d298c54074d00 (DEFAULT_TABLE table=default.Table_885d3fb0ab5f4d0a82972d94a34c2a88 [id=18d1df6385a24d26bfb44309aa58eaa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.367673 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07aef1eea7c04d9cbf35c82e45df7a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.368361 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e05628011634149b17d298c54074d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.371645 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cd6e4352cfc41e497dc58d158e24ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:31 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:38.374207 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 00e5c4bf71b44abca66b0e94995c533e (DEFAULT_TABLE table=default.Table_dcb6d015c5c543689fc117f18bdff260 [id=ff5165bdae864e59a46b71d1ca8d8611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.375022 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e5c4bf71b44abca66b0e94995c533e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:38.376396 14714 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.016s sys 0.030s
W20251211 08:57:38.376852 14714 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.651s user 0.016s sys 0.030s
I20251211 08:57:38.379084 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet dca342fd9de241b5884600199f06fbeb
I20251211 08:57:38.381484 14714 tablet_bootstrap.cc:492] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:38.381829 14714 ts_tablet_manager.cc:1403] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.668s user 0.021s sys 0.030s
W20251211 08:57:38.382982 14013 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45c7d841bd3643a799a8b96e82d6d675 consensus state has no leader
I20251211 08:57:38.383031 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 479d7fe06a284234aef420e3842f0eab with delete_type TABLET_DATA_DELETED (replaced by tablet f5c5b1aebde94dea81cb37a6a6998070) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:38.383868 14417 tablet_service.cc:1505] Processing CreateTablet for tablet 080d1700df4a4df09acda65c6d021d96 (DEFAULT_TABLE table=default.Table_26f4e336df6947149cb311c15bee56c1 [id=e647fb169f024aebb1f942fff0779130]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.383813 14714 raft_consensus.cc:359] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.384413 14714 raft_consensus.cc:385] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:38.384620 14714 raft_consensus.cc:740] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:38.384810 14417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080d1700df4a4df09acda65c6d021d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.385097 14714 consensus_queue.cc:260] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.385365 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 88860fa9d7e6414185f0a1cbc78b2571 (DEFAULT_TABLE table=default.Table_9d16e03fa298473cb4acdb00f6143d5e [id=ffc60f5888b54383922a7334cc661ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.370723 15083 raft_consensus.cc:493] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:38.387068 15083 raft_consensus.cc:3060] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:38.391342 14578 tablet_service.cc:1505] Processing CreateTablet for tablet f785a3cc2e8145eebae7e9c2885b0b1e (DEFAULT_TABLE table=default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.392484 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f785a3cc2e8145eebae7e9c2885b0b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.392982 14422 tablet_service.cc:1552] Processing DeleteTablet for tablet b8d753358f20476ba23d7d7b72e26e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:38.393913 14423 tablet_service.cc:1505] Processing CreateTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 (DEFAULT_TABLE table=default.Table_d40cb8a6b4974e2e9ff982e893a2c552 [id=d6acd0e03f2743349b345927f5ce7a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.394207 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet caf537c461c04a81b66f268ee98dc64e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:38.394817 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07aef1eea7c04d9cbf35c82e45df7a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.395066 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 8e05628011634149b17d298c54074d00 (DEFAULT_TABLE table=default.Table_885d3fb0ab5f4d0a82972d94a34c2a88 [id=18d1df6385a24d26bfb44309aa58eaa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.395522 14420 tablet_service.cc:1505] Processing CreateTablet for tablet cb2ff90b54544420b1aad9d2ab4132ff (DEFAULT_TABLE table=default.Table_25fc063a532541f5a7d6c73d8f2f0e33 [id=3c0379138b1c48e0a5400a465c4f372e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.395838 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e05628011634149b17d298c54074d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.396442 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb2ff90b54544420b1aad9d2ab4132ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.398118 14427 tablet_service.cc:1505] Processing CreateTablet for tablet f785a3cc2e8145eebae7e9c2885b0b1e (DEFAULT_TABLE table=default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.398916 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f785a3cc2e8145eebae7e9c2885b0b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.403128 14424 tablet_service.cc:1505] Processing CreateTablet for tablet c341b762afd8440bb7198ca134c8dffe (DEFAULT_TABLE table=default.Table_96d05b744df34836be68b4841e6c2a2b [id=86d8260b23ca4945acd9a70054b77c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.403545 14418 tablet_service.cc:1505] Processing CreateTablet for tablet e9905227c0a844d199362a442e7d630c (DEFAULT_TABLE table=default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b [id=45f67c67257e428f8ed2f16898ad5fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.403798 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c341b762afd8440bb7198ca134c8dffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.403692 14421 tablet_service.cc:1505] Processing CreateTablet for tablet 88860fa9d7e6414185f0a1cbc78b2571 (DEFAULT_TABLE table=default.Table_9d16e03fa298473cb4acdb00f6143d5e [id=ffc60f5888b54383922a7334cc661ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.404399 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9905227c0a844d199362a442e7d630c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.405385 14416 tablet_service.cc:1505] Processing CreateTablet for tablet f5c5b1aebde94dea81cb37a6a6998070 (DEFAULT_TABLE table=default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.406271 14416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c5b1aebde94dea81cb37a6a6998070. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.407195 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 165) took 2496 ms. Trace:
I20251211 08:57:38.407866 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88860fa9d7e6414185f0a1cbc78b2571. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.407666 14905 rpcz_store.cc:276] 1211 08:57:35.902461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.918313 (+ 15852us) service_pool.cc:225] Handling call
1211 08:57:37.905348 (+1987035us) ts_tablet_manager.cc:1151] Deleting tablet 479d7fe06a284234aef420e3842f0eab
1211 08:57:37.905377 (+ 29us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:38.039744 (+134367us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:38.182182 (+142438us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:38.398763 (+216581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1978446}
I20251211 08:57:38.409296 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 168) took 2463 ms. Trace:
I20251211 08:57:38.409700 14905 rpcz_store.cc:276] 1211 08:57:35.945776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.945896 (+ 120us) service_pool.cc:225] Handling call
1211 08:57:38.408920 (+2463024us) ts_tablet_manager.cc:1151] Deleting tablet 7b9b338a3f544ccb813247250b0f0eb7
1211 08:57:38.408975 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:38.409227 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2462528}
I20251211 08:57:38.411959 14429 tablet_service.cc:1552] Processing DeleteTablet for tablet 5daba165262e41af8d3481d3b4929092 with delete_type TABLET_DATA_DELETED (replaced by tablet da52cace2aab4f9dacef18ec5fa56aaf) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:38.412885 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 8e05628011634149b17d298c54074d00
I20251211 08:57:38.415364 14422 tablet_service.cc:1505] Processing CreateTablet for tablet 17ab48721a5c48a7b8a5550be658837d (DEFAULT_TABLE table=default.Table_f13b896c35754c0a94cc5b238778ecd2 [id=1b741ee1e2bd404f982e2c2aea8da710]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.416205 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17ab48721a5c48a7b8a5550be658837d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.419843 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 00e5c4bf71b44abca66b0e94995c533e (DEFAULT_TABLE table=default.Table_dcb6d015c5c543689fc117f18bdff260 [id=ff5165bdae864e59a46b71d1ca8d8611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.420747 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e5c4bf71b44abca66b0e94995c533e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.439565 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_460c971e87e84cd58224ed771dd7040e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:38.441660 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_460c971e87e84cd58224ed771dd7040e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:38.445497 15030 tablet_bootstrap.cc:654] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:38.454113 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88860fa9d7e6414185f0a1cbc78b2571. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.455938 15083 raft_consensus.cc:515] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.457690 15083 leader_election.cc:290] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:38.459066 14714 ts_tablet_manager.cc:1434] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.077s user 0.006s sys 0.000s
I20251211 08:57:38.460559 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f1a804079b754f0e84b85a175c773a3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:38.461836 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1a804079b754f0e84b85a175c773a3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:38.465093 14714 tablet_bootstrap.cc:492] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:38.471314 14584 tablet_service.cc:1505] Processing CreateTablet for tablet f5c5b1aebde94dea81cb37a6a6998070 (DEFAULT_TABLE table=default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.475463 14419 tablet_service.cc:1552] Processing DeleteTablet for tablet dca342fd9de241b5884600199f06fbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:38.478286 15192 raft_consensus.cc:493] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:38.478881 15122 raft_consensus.cc:493] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:38.478627 15192 raft_consensus.cc:515] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.479262 15122 raft_consensus.cc:515] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.480861 15122 leader_election.cc:290] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:38.481359 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 167) took 2536 ms. Trace:
I20251211 08:57:38.481626 14905 rpcz_store.cc:276] 1211 08:57:35.945475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:35.946467 (+ 992us) service_pool.cc:225] Handling call
1211 08:57:38.410739 (+2464272us) ts_tablet_manager.cc:1151] Deleting tablet a4e307e4d1994af0b99a88b3b06b73a4
1211 08:57:38.410784 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:38.481309 (+ 70525us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2463629}
I20251211 08:57:38.480722 15192 leader_election.cc:290] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:38.484922 14419 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e05628011634149b17d298c54074d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:38.486199 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 552de4c6d030418181f0400714d9ec31 (DEFAULT_TABLE table=default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.487119 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 552de4c6d030418181f0400714d9ec31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.324996 15302 raft_consensus.cc:493] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:38.495146 14904 tablet_replica.cc:333] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:38.495994 14904 raft_consensus.cc:2243] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:38.514891 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db302282bc3d4d8cb56d1cbb2f73c4fe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:38.516364 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet 9168007ca0584954a734e2c8f2e73f70 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:38.524484 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5daba165262e41af8d3481d3b4929092" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:38.525422 14605 raft_consensus.cc:1275] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Refusing update from remote peer b276f77f87254a5d909c5ab940a39256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:38.525693 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a1f12553934446bfe44f4ed41abf9a" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:38.526166 14607 raft_consensus.cc:3060] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:38.527808 14382 leader_election.cc:343] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:38.528319 14382 leader_election.cc:343] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:38.531701 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet dca342fd9de241b5884600199f06fbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:38.531991 14577 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e05628011634149b17d298c54074d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:38.534297 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 479d7fe06a284234aef420e3842f0eab (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]) successfully deleted
W20251211 08:57:38.535113 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 7b9b338a3f544ccb813247250b0f0eb7 because the tablet was not found. No further retry: Not found: Tablet not found: 7b9b338a3f544ccb813247250b0f0eb7
I20251211 08:57:38.537321 15152 raft_consensus.cc:493] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:38.537672 15152 raft_consensus.cc:515] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.538606 14278 tablet_service.cc:1505] Processing CreateTablet for tablet f785a3cc2e8145eebae7e9c2885b0b1e (DEFAULT_TABLE table=default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.539240 15152 leader_election.cc:290] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:38.539062 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 17ab48721a5c48a7b8a5550be658837d (DEFAULT_TABLE table=default.Table_f13b896c35754c0a94cc5b238778ecd2 [id=1b741ee1e2bd404f982e2c2aea8da710]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.539608 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f785a3cc2e8145eebae7e9c2885b0b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.492288 15302 raft_consensus.cc:3060] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:38.547783 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c5b1aebde94dea81cb37a6a6998070. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.548859 14607 raft_consensus.cc:2468] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:38.550246 14382 leader_election.cc:304] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:38.554577 15193 consensus_queue.cc:1048] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:38.555531 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 00e5c4bf71b44abca66b0e94995c533e (DEFAULT_TABLE table=default.Table_dcb6d015c5c543689fc117f18bdff260 [id=ff5165bdae864e59a46b71d1ca8d8611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.560849 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 170) took 2334 ms. Trace:
I20251211 08:57:38.561128 14905 rpcz_store.cc:276] 1211 08:57:36.226185 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.226317 (+ 132us) service_pool.cc:225] Handling call
1211 08:57:38.482255 (+2255938us) ts_tablet_manager.cc:1151] Deleting tablet 2016208a33dc4a5cadd7ced8e85ccc65
1211 08:57:38.560563 (+ 78308us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:38.560792 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2255367}
I20251211 08:57:38.562160 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 171) took 2195 ms. Trace:
I20251211 08:57:38.562407 14905 rpcz_store.cc:276] 1211 08:57:36.366374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.369883 (+ 3509us) service_pool.cc:225] Handling call
1211 08:57:38.561899 (+2192016us) ts_tablet_manager.cc:1151] Deleting tablet 9b5a4b6b1b28406c82e2acd9356b60d4
1211 08:57:38.561929 (+ 30us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:38.562118 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2191456}
I20251211 08:57:38.563349 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 172) took 2085 ms. Trace:
I20251211 08:57:38.563604 14905 rpcz_store.cc:276] 1211 08:57:36.477736 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.477966 (+ 230us) service_pool.cc:225] Handling call
1211 08:57:38.563073 (+2085107us) ts_tablet_manager.cc:1151] Deleting tablet 8ad600554c0946e0bcb0da6a42be2ffd
1211 08:57:38.563103 (+ 30us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:38.563309 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2084489}
I20251211 08:57:38.564405 14905 tablet_replica.cc:333] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:38.565019 14905 raft_consensus.cc:2243] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:38.565429 14905 raft_consensus.cc:2272] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:38.568003 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42720f5f5bf445f2b72e513ad4a6d95f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:38.568444 14449 raft_consensus.cc:2468] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:38.569159 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_1ab8b3b3713f428293902db0aa8cedb6" } new_table_name: "default.Table_70f3ebdd01dc4e75bc85ab29528865fc" modify_external_catalogs: true
I20251211 08:57:38.571576 15122 raft_consensus.cc:2804] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:38.571950 15122 raft_consensus.cc:697] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:38.572576 15122 consensus_queue.cc:237] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:38.575273 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 2016208a33dc4a5cadd7ced8e85ccc65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2016208a33dc4a5cadd7ced8e85ccc65 already in progress: creating tablet
W20251211 08:57:38.575785 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b5a4b6b1b28406c82e2acd9356b60d4 already in progress: creating tablet
W20251211 08:57:38.576318 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 8ad600554c0946e0bcb0da6a42be2ffd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ad600554c0946e0bcb0da6a42be2ffd already in progress: creating tablet
I20251211 08:57:38.578744 15152 consensus_queue.cc:1048] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:38.579751 14904 raft_consensus.cc:2272] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:38.585829 14009 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b" } modify_external_catalogs: true
I20251211 08:57:38.586287 14009 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b" } modify_external_catalogs: true
I20251211 08:57:38.586107 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_182b8f62776541cea88982532ffe9fcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:38.587579 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_182b8f62776541cea88982532ffe9fcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:38.591151 14010 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_39c89fc0d98a45ad831ac604f107a406" } new_table_name: "default.Table_b19d002dfb154df6aadac72b8859cce9" modify_external_catalogs: true
W20251211 08:57:38.591533 14542 consensus_peers.cc:597] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:38.593020 14542 leader_election.cc:304] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:38.593771 15152 raft_consensus.cc:2804] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:38.594096 15152 raft_consensus.cc:493] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:38.594385 15152 raft_consensus.cc:3060] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:38.596872 15192 raft_consensus.cc:493] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:38.597015 14905 ts_tablet_manager.cc:1916] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:38.597203 14714 tablet_bootstrap.cc:654] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:38.597285 15192 raft_consensus.cc:515] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.599027 15192 leader_election.cc:290] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:38.600153 15299 ts_tablet_manager.cc:1929] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:38.600502 15299 log.cc:1199] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/e6c4d4c99b3b496cab125e13153349cb
I20251211 08:57:38.601318 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a8437b8b464fb8ab02a9c552f81db3" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:38.601871 14605 raft_consensus.cc:2468] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:38.603622 14382 leader_election.cc:304] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:38.607944 15030 tablet_bootstrap.cc:492] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:38.609772 15030 ts_tablet_manager.cc:1403] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.429s user 0.011s sys 0.004s
I20251211 08:57:38.611907 15030 raft_consensus.cc:359] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.612411 15030 raft_consensus.cc:385] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:38.612625 15030 raft_consensus.cc:740] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:38.613121 15030 consensus_queue.cc:260] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.613729 15192 raft_consensus.cc:2804] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:38.614015 15192 raft_consensus.cc:493] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:38.614289 15192 raft_consensus.cc:3060] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:38.617901 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7ff0193278184b638a2f8e40fd323397" } modify_external_catalogs: true
I20251211 08:57:38.618327 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7ff0193278184b638a2f8e40fd323397" } modify_external_catalogs: true
I20251211 08:57:38.619143 15299 ts_tablet_manager.cc:1950] T e6c4d4c99b3b496cab125e13153349cb P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:38.621469 15299 tablet_replica.cc:333] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:38.621953 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet e6c4d4c99b3b496cab125e13153349cb (table default.Table_1ab8b3b3713f428293902db0aa8cedb6 [id=67d1517b43294838bda420b792edb9c5]) successfully deleted
I20251211 08:57:38.621837 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_0442aa229734421dab4cf2f058c9c5e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:38.623201 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0442aa229734421dab4cf2f058c9c5e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:38.624527 14904 ts_tablet_manager.cc:1916] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:38.631634 14271 tablet_service.cc:1505] Processing CreateTablet for tablet e9905227c0a844d199362a442e7d630c (DEFAULT_TABLE table=default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b [id=45f67c67257e428f8ed2f16898ad5fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.639978 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17ab48721a5c48a7b8a5550be658837d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.641891 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 9168007ca0584954a734e2c8f2e73f70
I20251211 08:57:38.643074 14013 catalog_manager.cc:6724] ReplaceTablet: tablet 9168007ca0584954a734e2c8f2e73f70 deleted and replaced by tablet f56fd93aeac6435ea84388ce05580113
W20251211 08:57:38.652745 14269 consensus_meta.cc:287] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b: Time spent flushing consensus metadata: real 0.783s user 0.009s sys 0.000s
I20251211 08:57:38.664402 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9905227c0a844d199362a442e7d630c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:38.666560 14273 consensus_meta.cc:287] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b: Time spent flushing consensus metadata: real 0.650s user 0.008s sys 0.000s
I20251211 08:57:38.682296 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet e9905227c0a844d199362a442e7d630c
I20251211 08:57:38.702219 14722 tablet_bootstrap.cc:492] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:38.702657 14722 ts_tablet_manager.cc:1403] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.757s user 0.013s sys 0.001s
I20251211 08:57:38.709329 15302 raft_consensus.cc:515] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.715744 14722 raft_consensus.cc:359] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.757078 14722 raft_consensus.cc:385] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:38.757496 14722 raft_consensus.cc:740] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:38.758193 14722 consensus_queue.cc:260] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.761106 14722 ts_tablet_manager.cc:1434] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.058s user 0.005s sys 0.000s
I20251211 08:57:38.795920 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet e6c4d4c99b3b496cab125e13153349cb (table default.Table_1ab8b3b3713f428293902db0aa8cedb6 [id=67d1517b43294838bda420b792edb9c5]) successfully deleted
I20251211 08:57:38.800901 14722 tablet_bootstrap.cc:492] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:38.810927 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.62:39015, user_credentials={real_user=slave}} blocked reactor thread for 47741.3us
I20251211 08:57:38.812363 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_d9c4c3852105418a914f5055b9b70072"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:38.813906 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9c4c3852105418a914f5055b9b70072 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:38.829839 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_939cb0831eff4bdebbb7fc6d0052a06a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:38.831329 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_939cb0831eff4bdebbb7fc6d0052a06a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:38.842981 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 9168007ca0584954a734e2c8f2e73f70 with delete_type TABLET_DATA_DELETED (replaced by tablet f56fd93aeac6435ea84388ce05580113) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:38.733501 14272 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46844 (request call id 177) took 1025 ms. Trace:
I20251211 08:57:38.847030 14272 rpcz_store.cc:276] 1211 08:57:37.707763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:37.712307 (+ 4544us) service_pool.cc:225] Handling call
1211 08:57:37.713349 (+ 1042us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1211 08:57:37.713414 (+ 65us) ts_tablet_manager.cc:682] Creating new metadata...
1211 08:57:38.238077 (+524663us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:38.732950 (+494873us) tablet_replica.cc:206] Creating consensus instance
1211 08:57:38.733415 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:57:38.775120 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e5c4bf71b44abca66b0e94995c533e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.848851 15152 raft_consensus.cc:515] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.850505 15152 leader_election.cc:290] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:38.851114 15030 ts_tablet_manager.cc:1434] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.241s user 0.019s sys 0.018s
I20251211 08:57:38.851977 15030 tablet_bootstrap.cc:492] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:38.778733 15302 leader_election.cc:290] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:38.791899 14265 tablet_service.cc:1505] Processing CreateTablet for tablet c341b762afd8440bb7198ca134c8dffe (DEFAULT_TABLE table=default.Table_96d05b744df34836be68b4841e6c2a2b [id=86d8260b23ca4945acd9a70054b77c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.855696 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c341b762afd8440bb7198ca134c8dffe. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:38.856926 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 91503.4us
I20251211 08:57:38.871731 14575 tablet_service.cc:1552] Processing DeleteTablet for tablet e9905227c0a844d199362a442e7d630c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:38.793684 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet dca342fd9de241b5884600199f06fbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:38.794791 14274 tablet_service.cc:1552] Processing DeleteTablet for tablet 9168007ca0584954a734e2c8f2e73f70 with delete_type TABLET_DATA_DELETED (replaced by tablet f56fd93aeac6435ea84388ce05580113) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:38.795166 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet e9905227c0a844d199362a442e7d630c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:38.797175 14268 tablet_service.cc:1505] Processing CreateTablet for tablet f5c5b1aebde94dea81cb37a6a6998070 (DEFAULT_TABLE table=default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:38.740583 14230 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.000s sys 0.020s
I20251211 08:57:38.794867 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 88860fa9d7e6414185f0a1cbc78b2571 (DEFAULT_TABLE table=default.Table_9d16e03fa298473cb4acdb00f6143d5e [id=ffc60f5888b54383922a7334cc661ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:38.803817 14269 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e05628011634149b17d298c54074d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:38.877326 14230 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.708s user 0.000s sys 0.022s
W20251211 08:57:38.878189 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 710269us
I20251211 08:57:38.652988 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet 9168007ca0584954a734e2c8f2e73f70 with delete_type TABLET_DATA_DELETED (replaced by tablet f56fd93aeac6435ea84388ce05580113) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:38.888275 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_6109f16f9985461fbb07b1d083bbb166" } modify_external_catalogs: true
I20251211 08:57:38.888839 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_6109f16f9985461fbb07b1d083bbb166" } modify_external_catalogs: true
I20251211 08:57:38.695073 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet e9905227c0a844d199362a442e7d630c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:38.716656 15192 raft_consensus.cc:515] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.776626 14714 tablet_bootstrap.cc:492] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:38.890749 14714 ts_tablet_manager.cc:1403] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.426s user 0.010s sys 0.017s
I20251211 08:57:38.891716 15192 leader_election.cc:290] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:38.778645 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caf537c461c04a81b66f268ee98dc64e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:38.892743 15030 tablet_bootstrap.cc:654] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:38.892788 14449 raft_consensus.cc:3060] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:38.795253 14385 leader_election.cc:343] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:38.895121 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 99946.4us
I20251211 08:57:38.869707 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42720f5f5bf445f2b72e513ad4a6d95f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:38.898295 14448 raft_consensus.cc:3060] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:38.898609 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a8437b8b464fb8ab02a9c552f81db3" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:38.899163 14605 raft_consensus.cc:3060] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:38.899755 14905 ts_tablet_manager.cc:1929] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:38.900154 14905 log.cc:1199] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/ad903727f79a4180adf047c92d981c7e
I20251211 08:57:38.900812 14905 ts_tablet_manager.cc:1950] T ad903727f79a4180adf047c92d981c7e P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:57:38.904906 14544 leader_election.cc:343] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:38.905725 14605 raft_consensus.cc:2468] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:38.905743 14714 raft_consensus.cc:359] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.906409 14714 raft_consensus.cc:385] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:38.907052 14382 leader_election.cc:304] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:38.907241 14448 raft_consensus.cc:2468] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:38.907859 15192 raft_consensus.cc:2804] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:38.908834 15192 raft_consensus.cc:697] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:38.909595 15192 consensus_queue.cc:237] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.910108 14449 raft_consensus.cc:2468] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:38.911595 14231 leader_election.cc:304] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:38.912842 15228 raft_consensus.cc:2804] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:38.913282 15228 raft_consensus.cc:697] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:38.902684 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 173) took 2182 ms. Trace:
I20251211 08:57:38.914095 15228 consensus_queue.cc:237] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.914592 14905 rpcz_store.cc:276] 1211 08:57:36.720152 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.720279 (+ 127us) service_pool.cc:225] Handling call
1211 08:57:38.564298 (+1844019us) ts_tablet_manager.cc:1151] Deleting tablet ad903727f79a4180adf047c92d981c7e
1211 08:57:38.564331 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:38.844849 (+280518us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:38.899544 (+ 54695us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:38.902630 (+ 3086us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1843555}
I20251211 08:57:38.915673 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet ad903727f79a4180adf047c92d981c7e (table default.Table_dda8864ad51542599828719c4cb52e5b [id=2c2c410df07d4e4cbc60e08dce22efe9]) successfully deleted
I20251211 08:57:38.914582 15299 raft_consensus.cc:2243] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:38.917240 15299 raft_consensus.cc:2272] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:38.918303 15030 tablet_bootstrap.cc:492] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:38.919526 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a1f12553934446bfe44f4ed41abf9a" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:38.919749 14905 tablet_replica.cc:333] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:38.906680 14714 raft_consensus.cc:740] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:38.920428 14905 raft_consensus.cc:2243] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:38.920859 14905 raft_consensus.cc:2272] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:38.920820 14714 consensus_queue.cc:260] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.922423 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c5b1aebde94dea81cb37a6a6998070. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.922461 14905 ts_tablet_manager.cc:1916] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:38.923949 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88860fa9d7e6414185f0a1cbc78b2571. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:38.924806 14714 ts_tablet_manager.cc:1434] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20251211 08:57:38.918691 15030 ts_tablet_manager.cc:1403] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.004s
I20251211 08:57:38.928351 15299 ts_tablet_manager.cc:1916] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:38.929054 15030 raft_consensus.cc:359] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.929626 15030 raft_consensus.cc:385] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:38.929867 15030 raft_consensus.cc:740] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:38.925724 14714 tablet_bootstrap.cc:492] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:38.937597 15319 consensus_queue.cc:1048] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:38.941134 14542 leader_election.cc:304] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:38.947340 15319 raft_consensus.cc:2804] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:38.949549 15319 raft_consensus.cc:697] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:38.950373 15319 consensus_queue.cc:237] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.954674 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42720f5f5bf445f2b72e513ad4a6d95f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:38.959424 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42720f5f5bf445f2b72e513ad4a6d95f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:38.967815 14299 raft_consensus.cc:1275] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Refusing update from remote peer b276f77f87254a5d909c5ab940a39256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:38.972239 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db302282bc3d4d8cb56d1cbb2f73c4fe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:38.972505 15152 raft_consensus.cc:493] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:38.973085 15152 raft_consensus.cc:515] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:38.973790 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5daba165262e41af8d3481d3b4929092" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:38.975144 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a8437b8b464fb8ab02a9c552f81db3" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:38.975450 15152 leader_election.cc:290] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:38.981833 14714 tablet_bootstrap.cc:654] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:39.002240 14006 master_service.cc:954] ReplaceTablet: received request to replace tablet a4e307e4d1994af0b99a88b3b06b73a4 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:39.002281 14266 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46844 (request call id 187) took 1033 ms. Trace:
I20251211 08:57:39.002588 14266 rpcz_store.cc:276] 1211 08:57:37.968396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.008930 (+ 40534us) service_pool.cc:225] Handling call
1211 08:57:38.010932 (+ 2002us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1211 08:57:38.011017 (+ 85us) ts_tablet_manager.cc:682] Creating new metadata...
1211 08:57:38.737327 (+726310us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:38.814612 (+ 77285us) tablet_replica.cc:206] Creating consensus instance
1211 08:57:39.002226 (+187614us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:57:38.947342 15030 consensus_queue.cc:260] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.015748 15030 ts_tablet_manager.cc:1434] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.089s user 0.005s sys 0.000s
I20251211 08:57:39.016649 15030 tablet_bootstrap.cc:492] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:38.996155 15329 tablet.cc:1722] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:38.984761 15327 tablet.cc:1722] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:38.992724 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5daba165262e41af8d3481d3b4929092" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:39.026474 14449 raft_consensus.cc:2468] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:39.027706 14542 leader_election.cc:304] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:39.034967 15152 raft_consensus.cc:2804] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:39.035108 14722 tablet_bootstrap.cc:654] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:39.035410 15152 raft_consensus.cc:493] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:39.035807 15152 raft_consensus.cc:3060] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:39.044699 14005 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet d4f826d079774152a3b66bfb81f990a8 (table default.Table_4ab19ee3f1f3473fa3d320e7841a201c [id=4208ab42111742f38028caeb7913a1c7]). Expected version 1 got 0
I20251211 08:57:39.047407 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet f785a3cc2e8145eebae7e9c2885b0b1e
I20251211 08:57:39.047577 14299 raft_consensus.cc:3060] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:39.050174 14009 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 383716a1f1f847ccbea542eca04790c9 (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]). Expected version 2 got 0
I20251211 08:57:39.051510 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5daba165262e41af8d3481d3b4929092" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:39.051831 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 51c3f5eaef784e868e565f81bd51a5e4 (DEFAULT_TABLE table=default.Table_7343d67071c54141af5a57615f82221e [id=fe5786801bab48888480739b8ff266e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.052817 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c3f5eaef784e868e565f81bd51a5e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:39.053809 14544 consensus_peers.cc:597] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:39.054006 15030 tablet_bootstrap.cc:654] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:39.054598 14544 leader_election.cc:343] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.060762 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet f785a3cc2e8145eebae7e9c2885b0b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:39.066242 14544 consensus_peers.cc:597] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6c4d4c99b3b496cab125e13153349cb. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:39.067122 14544 leader_election.cc:343] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.068475 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet f785a3cc2e8145eebae7e9c2885b0b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:39.072371 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_3ec948949bdb4957b702fc9f5b18207a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:39.073640 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet f3403116517f45c59f87fa5bda0b4071
I20251211 08:57:39.073558 15152 raft_consensus.cc:515] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:39.074232 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ec948949bdb4957b702fc9f5b18207a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:39.076220 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet f3403116517f45c59f87fa5bda0b4071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:39.076090 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_5f7ac9a06e714b2bb4790efd948dc9ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:39.077826 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f7ac9a06e714b2bb4790efd948dc9ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:39.078608 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet f3403116517f45c59f87fa5bda0b4071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:39.083688 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 51c3f5eaef784e868e565f81bd51a5e4 (DEFAULT_TABLE table=default.Table_7343d67071c54141af5a57615f82221e [id=fe5786801bab48888480739b8ff266e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.084559 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c3f5eaef784e868e565f81bd51a5e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:39.081171 14230 leader_election.cc:343] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: caf537c461c04a81b66f268ee98dc64e
I20251211 08:57:39.088366 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5daba165262e41af8d3481d3b4929092" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:39.088860 14449 raft_consensus.cc:3060] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:39.088733 14580 tablet_service.cc:1505] Processing CreateTablet for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (DEFAULT_TABLE table=default.Table_fd201f899f064ec0b339bd8b7679fefc [id=f74e074787144a21ad31d4d89cc509c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.090224 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.088616 14424 tablet_service.cc:1505] Processing CreateTablet for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (DEFAULT_TABLE table=default.Table_fd201f899f064ec0b339bd8b7679fefc [id=f74e074787144a21ad31d4d89cc509c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.099730 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.103839 14449 raft_consensus.cc:2468] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:39.104827 14542 leader_election.cc:304] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:39.105643 15318 raft_consensus.cc:2804] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:39.106060 15318 raft_consensus.cc:697] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:39.107301 15318 consensus_queue.cc:237] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.111156 15192 raft_consensus.cc:493] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:39.111817 14722 tablet_bootstrap.cc:492] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:39.111625 15192 raft_consensus.cc:515] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.112200 14722 ts_tablet_manager.cc:1403] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.312s user 0.005s sys 0.004s
I20251211 08:57:39.114229 15192 leader_election.cc:290] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:39.114157 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 (DEFAULT_TABLE table=default.Table_03bee763242749cfa5fb0fe7d7e7fb7f [id=209008ecbdd4441989221f0f956a87b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.115145 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1b011cb7c647f79bb16ce1c92bcff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.115429 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78c6a703e1e44f41931fe465def01bf6" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:39.116242 14605 raft_consensus.cc:2468] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:39.117066 14382 leader_election.cc:304] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:39.117869 15192 raft_consensus.cc:2804] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:39.118191 15192 raft_consensus.cc:493] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:39.118500 15192 raft_consensus.cc:3060] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:39.119354 14011 catalog_manager.cc:5654] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:39.120645 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 (DEFAULT_TABLE table=default.Table_03bee763242749cfa5fb0fe7d7e7fb7f [id=209008ecbdd4441989221f0f956a87b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.120805 14584 tablet_service.cc:1505] Processing CreateTablet for tablet f3403116517f45c59f87fa5bda0b4071 (DEFAULT_TABLE table=default.Table_6109f16f9985461fbb07b1d083bbb166 [id=58f79d91f6e64d7980b96715d3dd25e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.121096 14011 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 383716a1f1f847ccbea542eca04790c9 (table default.Table_38d82f258eab426daa74a3a31f588767 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a]). Expected version 2 got 0
I20251211 08:57:39.121564 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1b011cb7c647f79bb16ce1c92bcff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.121773 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3403116517f45c59f87fa5bda0b4071. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.120760 14426 tablet_service.cc:1505] Processing CreateTablet for tablet f3403116517f45c59f87fa5bda0b4071 (DEFAULT_TABLE table=default.Table_6109f16f9985461fbb07b1d083bbb166 [id=58f79d91f6e64d7980b96715d3dd25e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.123333 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3403116517f45c59f87fa5bda0b4071. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.121758 14011 catalog_manager.cc:5654] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:39.125684 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78c6a703e1e44f41931fe465def01bf6" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:39.126883 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8a8437b8b464fb8ab02a9c552f81db3" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:39.127326 14385 leader_election.cc:343] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.128249 15193 consensus_queue.cc:1048] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:39.128955 14385 leader_election.cc:343] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.129289 14385 leader_election.cc:304] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
W20251211 08:57:39.130151 14385 leader_election.cc:343] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.130321 15193 raft_consensus.cc:2749] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:39.130462 14385 leader_election.cc:304] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:39.140471 15083 raft_consensus.cc:2749] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:39.141335 14385 leader_election.cc:343] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.143564 14722 raft_consensus.cc:359] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.144125 14722 raft_consensus.cc:385] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.144337 14722 raft_consensus.cc:740] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
W20251211 08:57:39.144953 14544 leader_election.cc:343] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.144829 14722 consensus_queue.cc:260] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.147794 15319 consensus_queue.cc:1048] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:39.163308 15152 leader_election.cc:290] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:39.165747 14714 tablet_bootstrap.cc:492] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:39.166158 14714 ts_tablet_manager.cc:1403] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.241s user 0.012s sys 0.000s
I20251211 08:57:39.168498 14714 raft_consensus.cc:359] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.168794 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5daba165262e41af8d3481d3b4929092" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:39.170251 14904 ts_tablet_manager.cc:1929] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:39.170580 14904 log.cc:1199] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/afe339c4cc95490683618e18a3da252b
W20251211 08:57:39.171463 14544 leader_election.cc:343] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.171759 14714 raft_consensus.cc:385] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.172013 14714 raft_consensus.cc:740] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:39.173731 14714 consensus_queue.cc:260] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.200101 14905 ts_tablet_manager.cc:1929] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:39.200507 14905 log.cc:1199] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/996c85729acd4418be1763b73cbf7d2d
I20251211 08:57:39.201308 14905 ts_tablet_manager.cc:1950] T 996c85729acd4418be1763b73cbf7d2d P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:39.206005 15299 ts_tablet_manager.cc:1929] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:39.206893 14904 ts_tablet_manager.cc:1950] T afe339c4cc95490683618e18a3da252b P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:39.180069 14722 ts_tablet_manager.cc:1434] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.068s user 0.004s sys 0.002s
I20251211 08:57:39.182274 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e307e4d1994af0b99a88b3b06b73a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec7694087145d8a49dd93904f95980) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:39.183216 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 (DEFAULT_TABLE table=default.Table_03bee763242749cfa5fb0fe7d7e7fb7f [id=209008ecbdd4441989221f0f956a87b6]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:39.207734 14385 leader_election.cc:343] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.212426 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1b011cb7c647f79bb16ce1c92bcff5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:39.212651 14385 leader_election.cc:343] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.213408 15299 log.cc:1199] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/afe339c4cc95490683618e18a3da252b
I20251211 08:57:39.214079 15299 ts_tablet_manager.cc:1950] T afe339c4cc95490683618e18a3da252b P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:39.184116 14267 tablet_service.cc:1505] Processing CreateTablet for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (DEFAULT_TABLE table=default.Table_fd201f899f064ec0b339bd8b7679fefc [id=f74e074787144a21ad31d4d89cc509c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.215075 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 184) took 1375 ms. Trace:
I20251211 08:57:39.215545 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.216020 15030 tablet_bootstrap.cc:492] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:39.216468 15030 ts_tablet_manager.cc:1403] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.200s user 0.006s sys 0.005s
I20251211 08:57:39.215344 14904 rpcz_store.cc:276] 1211 08:57:37.839291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:37.840657 (+ 1366us) service_pool.cc:225] Handling call
1211 08:57:38.495009 (+654352us) ts_tablet_manager.cc:1151] Deleting tablet afe339c4cc95490683618e18a3da252b
1211 08:57:38.495044 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:38.942422 (+447378us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.170044 (+227622us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.215027 (+ 44983us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82238,"tablet-delete.queue_time_us":653919}
I20251211 08:57:39.218636 15030 raft_consensus.cc:359] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.219214 15030 raft_consensus.cc:385] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.219456 15030 raft_consensus.cc:740] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:39.214197 15192 raft_consensus.cc:515] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.223987 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 196) took 1381 ms. Trace:
I20251211 08:57:39.224246 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 185) took 1370 ms. Trace:
I20251211 08:57:39.224583 14904 rpcz_store.cc:276] 1211 08:57:37.853686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:37.862852 (+ 9166us) service_pool.cc:225] Handling call
1211 08:57:39.223346 (+1360494us) ts_tablet_manager.cc:1151] Deleting tablet c341b762afd8440bb7198ca134c8dffe
1211 08:57:39.223405 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.223623 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1360051}
I20251211 08:57:39.224345 15299 rpcz_store.cc:276] 1211 08:57:37.842188 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:37.842298 (+ 110us) service_pool.cc:225] Handling call
1211 08:57:38.621334 (+779036us) ts_tablet_manager.cc:1151] Deleting tablet afe339c4cc95490683618e18a3da252b
1211 08:57:38.621368 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.180386 (+559018us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.205748 (+ 25362us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.223935 (+ 18187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19581,"tablet-delete.queue_time_us":778586}
I20251211 08:57:39.211158 14714 ts_tablet_manager.cc:1434] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.045s user 0.001s sys 0.004s
I20251211 08:57:39.184671 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet f3403116517f45c59f87fa5bda0b4071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:39.226207 14714 tablet_bootstrap.cc:492] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:39.230221 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78c6a703e1e44f41931fe465def01bf6" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:39.230199 15030 consensus_queue.cc:260] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:39.231406 14385 leader_election.cc:343] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.233219 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78c6a703e1e44f41931fe465def01bf6" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:39.233892 14605 raft_consensus.cc:3060] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:39.234758 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 198) took 1382 ms. Trace:
I20251211 08:57:39.235174 15299 rpcz_store.cc:276] 1211 08:57:37.852336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:37.852451 (+ 115us) service_pool.cc:225] Handling call
1211 08:57:39.234383 (+1381932us) ts_tablet_manager.cc:1151] Deleting tablet c341b762afd8440bb7198ca134c8dffe
1211 08:57:39.234426 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.234700 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1381449}
I20251211 08:57:39.182053 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet f785a3cc2e8145eebae7e9c2885b0b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:39.186473 14265 tablet_service.cc:1552] Processing DeleteTablet for tablet caf537c461c04a81b66f268ee98dc64e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:39.188071 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 51c3f5eaef784e868e565f81bd51a5e4 (DEFAULT_TABLE table=default.Table_7343d67071c54141af5a57615f82221e [id=fe5786801bab48888480739b8ff266e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.189109 14269 tablet_service.cc:1505] Processing CreateTablet for tablet f3403116517f45c59f87fa5bda0b4071 (DEFAULT_TABLE table=default.Table_6109f16f9985461fbb07b1d083bbb166 [id=58f79d91f6e64d7980b96715d3dd25e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.238729 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3403116517f45c59f87fa5bda0b4071. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.239431 15299 tablet_replica.cc:333] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:39.239959 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 188) took 1204 ms. Trace:
I20251211 08:57:39.240250 15299 raft_consensus.cc:2243] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:39.240264 14904 rpcz_store.cc:276] 1211 08:57:38.035649 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.035765 (+ 116us) service_pool.cc:225] Handling call
1211 08:57:39.239656 (+1203891us) ts_tablet_manager.cc:1151] Deleting tablet 81cfb3e0baef45ddab4a77a4e0790085
1211 08:57:39.239697 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.239903 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085136}
I20251211 08:57:39.241191 15299 raft_consensus.cc:2272] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:39.241405 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 174) took 2247 ms. Trace:
I20251211 08:57:39.241724 14905 rpcz_store.cc:276] 1211 08:57:36.994298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:36.994454 (+ 156us) service_pool.cc:225] Handling call
1211 08:57:38.919619 (+1925165us) ts_tablet_manager.cc:1151] Deleting tablet 996c85729acd4418be1763b73cbf7d2d
1211 08:57:38.919658 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:38.942667 (+ 23009us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.199829 (+257162us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.241349 (+ 41520us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1924638}
I20251211 08:57:39.190150 14273 tablet_service.cc:1505] Processing CreateTablet for tablet c7c04583bef74d84a49cdcd8a1305a88 (DEFAULT_TABLE table=default.Table_c0e87778556043e9ad8242434ab127c3 [id=202795cefb8a49dfa9c9353f60bc4120]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.243075 15299 ts_tablet_manager.cc:1916] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:39.243853 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c04583bef74d84a49cdcd8a1305a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.245880 14605 raft_consensus.cc:2468] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:39.246935 14382 leader_election.cc:304] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:39.247646 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c3f5eaef784e868e565f81bd51a5e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.247677 14714 tablet_bootstrap.cc:654] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:39.247725 15122 raft_consensus.cc:2804] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:39.248129 15122 raft_consensus.cc:697] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:39.248898 15122 consensus_queue.cc:237] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.250101 14905 tablet_replica.cc:333] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:39.250795 14905 raft_consensus.cc:2243] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:39.253479 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 190) took 1090 ms. Trace:
I20251211 08:57:39.253969 15192 leader_election.cc:290] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:39.253788 14904 rpcz_store.cc:276] 1211 08:57:38.162895 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.162994 (+ 99us) service_pool.cc:225] Handling call
1211 08:57:39.253254 (+1090260us) ts_tablet_manager.cc:1151] Deleting tablet 2cd6e4352cfc41e497dc58d158e24ff2
1211 08:57:39.253295 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.253432 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1072169}
I20251211 08:57:39.258932 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 192) took 1089 ms. Trace:
I20251211 08:57:39.259220 14904 rpcz_store.cc:276] 1211 08:57:38.169351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.182109 (+ 12758us) service_pool.cc:225] Handling call
1211 08:57:39.258603 (+1076494us) ts_tablet_manager.cc:1151] Deleting tablet 4eec7694087145d8a49dd93904f95980
1211 08:57:39.258645 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.258884 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075908}
I20251211 08:57:39.260876 14905 pending_rounds.cc:70] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:57:39.261083 14905 pending_rounds.cc:77] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256401316958208 op_type: NO_OP noop_request { }
I20251211 08:57:39.261391 14905 raft_consensus.cc:2889] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:39.261698 14905 raft_consensus.cc:2272] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:39.263343 14905 ts_tablet_manager.cc:1916] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:39.268474 14714 tablet_bootstrap.cc:492] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:39.268997 14714 ts_tablet_manager.cc:1403] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.001s
I20251211 08:57:39.271153 14904 tablet_replica.cc:333] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:39.271840 14904 raft_consensus.cc:2243] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:39.271443 14714 raft_consensus.cc:359] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.272145 15030 ts_tablet_manager.cc:1434] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.055s user 0.005s sys 0.000s
I20251211 08:57:39.272183 14714 raft_consensus.cc:385] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.272275 14904 raft_consensus.cc:2272] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:39.273061 15030 tablet_bootstrap.cc:492] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:39.273927 14904 ts_tablet_manager.cc:1916] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:39.272496 14714 raft_consensus.cc:740] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:39.283393 14714 consensus_queue.cc:260] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.285633 14714 ts_tablet_manager.cc:1434] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20251211 08:57:39.286631 14714 tablet_bootstrap.cc:492] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:39.287639 14722 tablet_bootstrap.cc:492] T 40a1f12553934446bfe44f4ed41abf9a P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:39.309695 15299 ts_tablet_manager.cc:1929] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:39.310097 15299 log.cc:1199] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/81cfb3e0baef45ddab4a77a4e0790085
I20251211 08:57:39.310812 15299 ts_tablet_manager.cc:1950] T 81cfb3e0baef45ddab4a77a4e0790085 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:39.313925 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 200) took 1267 ms. Trace:
I20251211 08:57:39.314307 15299 rpcz_store.cc:276] 1211 08:57:38.046040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.046150 (+ 110us) service_pool.cc:225] Handling call
1211 08:57:39.239293 (+1193143us) ts_tablet_manager.cc:1151] Deleting tablet 81cfb3e0baef45ddab4a77a4e0790085
1211 08:57:39.239329 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.276846 (+ 37517us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.309460 (+ 32614us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.313873 (+ 4413us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1125672}
I20251211 08:57:39.312069 14714 tablet_bootstrap.cc:654] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:39.318032 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 204) took 1219 ms. Trace:
I20251211 08:57:39.318374 15299 rpcz_store.cc:276] 1211 08:57:38.098948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.107878 (+ 8930us) service_pool.cc:225] Handling call
1211 08:57:39.317805 (+1209927us) ts_tablet_manager.cc:1151] Deleting tablet 1976d584cc0a450ab809ead258a251e2
1211 08:57:39.317842 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.317986 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1203645}
I20251211 08:57:39.321794 15299 tablet_replica.cc:333] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:39.323123 15299 raft_consensus.cc:2243] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:39.324556 15299 raft_consensus.cc:2272] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:39.327203 14385 consensus_peers.cc:597] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:39.329260 15299 ts_tablet_manager.cc:1916] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:39.331741 15318 raft_consensus.cc:493] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:39.332088 15318 raft_consensus.cc:515] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.333691 15318 leader_election.cc:290] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:39.335315 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f826d079774152a3b66bfb81f990a8" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:39.335911 14449 raft_consensus.cc:2468] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:39.337054 14542 leader_election.cc:304] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:39.338747 15318 raft_consensus.cc:2804] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:39.338905 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f826d079774152a3b66bfb81f990a8" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:39.339177 15318 raft_consensus.cc:493] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:39.339471 15318 raft_consensus.cc:3060] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:39.339927 14544 leader_election.cc:343] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.342013 14714 tablet_bootstrap.cc:492] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:39.342353 14714 ts_tablet_manager.cc:1403] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.000s
I20251211 08:57:39.344242 14714 raft_consensus.cc:359] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.344774 14714 raft_consensus.cc:385] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.344988 14714 raft_consensus.cc:740] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:39.345572 14714 consensus_queue.cc:260] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.347988 14714 ts_tablet_manager.cc:1434] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20251211 08:57:39.349298 14714 tablet_bootstrap.cc:492] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:39.351295 15318 raft_consensus.cc:515] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.354091 14714 tablet_bootstrap.cc:654] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:39.356796 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f826d079774152a3b66bfb81f990a8" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:39.357229 14449 raft_consensus.cc:3060] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:39.362373 14449 raft_consensus.cc:2468] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:39.363283 14542 leader_election.cc:304] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:39.363994 15152 raft_consensus.cc:2804] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:39.364444 15152 raft_consensus.cc:697] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:39.365216 15152 consensus_queue.cc:237] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.367327 15299 ts_tablet_manager.cc:1929] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:39.367230 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f826d079774152a3b66bfb81f990a8" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:39.367671 15299 log.cc:1199] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/b8d753358f20476ba23d7d7b72e26e74
I20251211 08:57:39.368265 15299 ts_tablet_manager.cc:1950] T b8d753358f20476ba23d7d7b72e26e74 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
W20251211 08:57:39.368291 14544 leader_election.cc:343] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.369422 15318 leader_election.cc:290] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:39.374203 14714 tablet_bootstrap.cc:492] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:39.374620 14714 ts_tablet_manager.cc:1403] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
I20251211 08:57:39.375638 15343 tablet.cc:1722] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:39.377463 14714 raft_consensus.cc:359] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.378015 14714 raft_consensus.cc:385] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.378238 14714 raft_consensus.cc:740] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:39.378753 14714 consensus_queue.cc:260] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.380230 15299 tablet_replica.cc:333] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:39.381268 15299 raft_consensus.cc:2243] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:39.387677 14575 tablet_service.cc:1505] Processing CreateTablet for tablet c7c04583bef74d84a49cdcd8a1305a88 (DEFAULT_TABLE table=default.Table_c0e87778556043e9ad8242434ab127c3 [id=202795cefb8a49dfa9c9353f60bc4120]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.383909 15299 raft_consensus.cc:2272] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:39.388742 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c04583bef74d84a49cdcd8a1305a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.388484 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 996c85729acd4418be1763b73cbf7d2d (table default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]) successfully deleted
I20251211 08:57:39.383553 14714 ts_tablet_manager.cc:1434] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251211 08:57:39.390575 14427 tablet_service.cc:1505] Processing CreateTablet for tablet c7c04583bef74d84a49cdcd8a1305a88 (DEFAULT_TABLE table=default.Table_c0e87778556043e9ad8242434ab127c3 [id=202795cefb8a49dfa9c9353f60bc4120]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.390937 14714 tablet_bootstrap.cc:492] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:39.391518 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c04583bef74d84a49cdcd8a1305a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.391441 14580 tablet_service.cc:1505] Processing CreateTablet for tablet f2f095b59ecf44ffabaa0ac22212938b (DEFAULT_TABLE table=default.Table_fee4bd06cd7e4df08fa4bafdd291a3d4 [id=e754a76685124291a154d2a35952abd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.391986 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet afe339c4cc95490683618e18a3da252b (table default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]) successfully deleted
I20251211 08:57:39.392424 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f095b59ecf44ffabaa0ac22212938b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:39.392738 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet c341b762afd8440bb7198ca134c8dffe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c341b762afd8440bb7198ca134c8dffe already in progress: creating tablet
I20251211 08:57:39.393472 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 81cfb3e0baef45ddab4a77a4e0790085 (table default.Table_02e79cde30df4a2d8e81657161c2a56e [id=8ac1575fb83e405493d250160ca6ac24]) successfully deleted
W20251211 08:57:39.394141 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 1976d584cc0a450ab809ead258a251e2 because the tablet was not found. No further retry: Not found: Tablet not found: 1976d584cc0a450ab809ead258a251e2
W20251211 08:57:39.394726 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 4eec7694087145d8a49dd93904f95980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4eec7694087145d8a49dd93904f95980 already in progress: creating tablet
W20251211 08:57:39.395354 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 479d7fe06a284234aef420e3842f0eab because the tablet was not found. No further retry: Not found: Tablet not found: 479d7fe06a284234aef420e3842f0eab
I20251211 08:57:39.395886 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet b8d753358f20476ba23d7d7b72e26e74 (table default.Table_6cbe43becdc846bcab4b1233f65cda0c [id=8a6e51e5adbf4ca39770244bfdc843bc]) successfully deleted
I20251211 08:57:39.396548 14714 tablet_bootstrap.cc:654] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:39.397436 13999 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 578404732eef4e51b66b6aebffa87cdb (table default.Table_e66c701250064c388e221a300dac3dd0 [id=26c0a9d9b1f84372b18a54f045eb768c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 578404732eef4e51b66b6aebffa87cdb consensus state has no leader
I20251211 08:57:39.397956 15299 ts_tablet_manager.cc:1916] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:39.411139 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d8d84e594ea47f3bc9127cc587e8aa7 consensus state has no leader
I20251211 08:57:39.414106 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_111bf58f55b84e8088d40992e55dd3f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:39.416319 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_111bf58f55b84e8088d40992e55dd3f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:39.418596 14010 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8a8437b8b464fb8ab02a9c552f81db3 consensus state has no leader
I20251211 08:57:39.422189 14427 tablet_service.cc:1505] Processing CreateTablet for tablet f2f095b59ecf44ffabaa0ac22212938b (DEFAULT_TABLE table=default.Table_fee4bd06cd7e4df08fa4bafdd291a3d4 [id=e754a76685124291a154d2a35952abd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.423061 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f095b59ecf44ffabaa0ac22212938b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.403465 14268 tablet_service.cc:1505] Processing CreateTablet for tablet f2f095b59ecf44ffabaa0ac22212938b (DEFAULT_TABLE table=default.Table_fee4bd06cd7e4df08fa4bafdd291a3d4 [id=e754a76685124291a154d2a35952abd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.430393 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f095b59ecf44ffabaa0ac22212938b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.441310 15030 tablet_bootstrap.cc:654] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:39.447120 15299 ts_tablet_manager.cc:1929] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:39.447474 15299 log.cc:1199] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/caf537c461c04a81b66f268ee98dc64e
I20251211 08:57:39.448077 15299 ts_tablet_manager.cc:1950] T caf537c461c04a81b66f268ee98dc64e P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:39.449990 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 215) took 1056 ms. Trace:
I20251211 08:57:39.450371 15299 rpcz_store.cc:276] 1211 08:57:38.393663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.394108 (+ 445us) service_pool.cc:225] Handling call
1211 08:57:39.380074 (+985966us) ts_tablet_manager.cc:1151] Deleting tablet caf537c461c04a81b66f268ee98dc64e
1211 08:57:39.380121 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.424051 (+ 43930us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.446891 (+ 22840us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.449940 (+ 3049us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":985522}
I20251211 08:57:39.451783 15299 tablet_replica.cc:333] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:39.454577 15122 consensus_queue.cc:1048] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:39.464747 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet caf537c461c04a81b66f268ee98dc64e (table default.Table_ac11e9583ef64c1bbae0b2940213b819 [id=8c183fcd39ae4bbeb0a05bfdd42e755e]) successfully deleted
I20251211 08:57:39.467340 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ed34b7fd8ce54c44a953a28d2fe518db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:39.467893 14005 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 30) took 951 ms (client timeout 1000 ms). Trace:
W20251211 08:57:39.468541 14005 rpcz_store.cc:269] 1211 08:57:38.516146 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.518874 (+ 2728us) service_pool.cc:225] Handling call
1211 08:57:39.467707 (+948833us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9402,"reported_tablets":7}
W20251211 08:57:39.469034 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed34b7fd8ce54c44a953a28d2fe518db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:39.470409 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 42720f5f5bf445f2b72e513ad4a6d95f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:39.470930 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet afe339c4cc95490683618e18a3da252b with delete_type TABLET_DATA_DELETED (replaced by tablet e36c6e2868e143be8e7f72307202a268) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:39.471470 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet db302282bc3d4d8cb56d1cbb2f73c4fe with delete_type TABLET_DATA_DELETED (replaced by tablet f785a3cc2e8145eebae7e9c2885b0b1e) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:39.482123 14385 consensus_peers.cc:597] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:39.490489 14722 tablet_bootstrap.cc:654] T 40a1f12553934446bfe44f4ed41abf9a P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:39.492486 14905 ts_tablet_manager.cc:1929] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:39.492852 14905 log.cc:1199] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/e6c4d4c99b3b496cab125e13153349cb
I20251211 08:57:39.493468 14905 ts_tablet_manager.cc:1950] T e6c4d4c99b3b496cab125e13153349cb P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:39.494727 15193 raft_consensus.cc:493] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:39.495116 15193 raft_consensus.cc:515] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.496778 15193 leader_election.cc:290] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:39.500523 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "174fefd70bcc41418a77a5716f97cd5f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:39.501788 14008 master_service.cc:954] ReplaceTablet: received request to replace tablet 9ccfcacb5b294a628926d34cd31b137e from {username='slave'} at 127.0.0.1:55614
W20251211 08:57:39.501861 14385 leader_election.cc:343] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.501853 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_460c971e87e84cd58224ed771dd7040e" } modify_external_catalogs: true
I20251211 08:57:39.502447 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_460c971e87e84cd58224ed771dd7040e" } modify_external_catalogs: true
I20251211 08:57:39.503329 14012 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d8ac34f0cab345dd9b113ef1447ea361" } modify_external_catalogs: true
I20251211 08:57:39.503759 14012 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d8ac34f0cab345dd9b113ef1447ea361" } modify_external_catalogs: true
I20251211 08:57:39.501358 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet afe339c4cc95490683618e18a3da252b (table default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]) successfully deleted
W20251211 08:57:39.511379 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet c341b762afd8440bb7198ca134c8dffe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c341b762afd8440bb7198ca134c8dffe already in progress: creating tablet
W20251211 08:57:39.511927 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 81cfb3e0baef45ddab4a77a4e0790085 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81cfb3e0baef45ddab4a77a4e0790085 already in progress: creating tablet
W20251211 08:57:39.512482 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 2cd6e4352cfc41e497dc58d158e24ff2 because the tablet was not found. No further retry: Not found: Tablet not found: 2cd6e4352cfc41e497dc58d158e24ff2
W20251211 08:57:39.513051 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 4eec7694087145d8a49dd93904f95980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4eec7694087145d8a49dd93904f95980 already in progress: creating tablet
I20251211 08:57:39.526389 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 033c2c016f8f4c01adc8bdbf2885e394 (DEFAULT_TABLE table=default.Table_1ce47a525fa9427198a4b8c589d662c7 [id=c7c35cc13c2840fe969a649388e5d250]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.526876 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 111adfd1ac1a42bdafa9416594e7e932 (DEFAULT_TABLE table=default.Table_c65937239ed54bb892938599e5f28043 [id=03d8c8d3f3844ed19a97971ed884317b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.526970 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 111adfd1ac1a42bdafa9416594e7e932 (DEFAULT_TABLE table=default.Table_c65937239ed54bb892938599e5f28043 [id=03d8c8d3f3844ed19a97971ed884317b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.527305 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 033c2c016f8f4c01adc8bdbf2885e394. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.527819 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 111adfd1ac1a42bdafa9416594e7e932. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.527838 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 111adfd1ac1a42bdafa9416594e7e932. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.531327 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 033c2c016f8f4c01adc8bdbf2885e394 (DEFAULT_TABLE table=default.Table_1ce47a525fa9427198a4b8c589d662c7 [id=c7c35cc13c2840fe969a649388e5d250]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.532213 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 033c2c016f8f4c01adc8bdbf2885e394. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.526389 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 0fdc020f21aa4f35880a435d7cfb605d (DEFAULT_TABLE table=default.Table_ccad8285d34949138091df8e2fafd435 [id=c747e05fbb284dd396393d08f7383388]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.534442 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdc020f21aa4f35880a435d7cfb605d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.534938 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "174fefd70bcc41418a77a5716f97cd5f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:39.537056 15299 raft_consensus.cc:2243] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:39.537513 15299 raft_consensus.cc:2272] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:39.539186 15299 ts_tablet_manager.cc:1916] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:39.539299 14382 leader_election.cc:343] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.539649 14382 leader_election.cc:304] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:39.540629 15193 raft_consensus.cc:2749] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:39.558233 14000 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 44689.1us
I20251211 08:57:39.564205 14005 master_service.cc:954] ReplaceTablet: received request to replace tablet da57b72207414512b9744b283bc869fe from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:39.566205 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 111adfd1ac1a42bdafa9416594e7e932 (DEFAULT_TABLE table=default.Table_c65937239ed54bb892938599e5f28043 [id=03d8c8d3f3844ed19a97971ed884317b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.567294 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 111adfd1ac1a42bdafa9416594e7e932. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.567642 14576 tablet_service.cc:1505] Processing CreateTablet for tablet ed0e90efe651499c929b1fb992712b96 (DEFAULT_TABLE table=default.Table_eb3b1d93a3ec4891a9d33b131661fd65 [id=46237116be8e46a68f51e20e008f374b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.568634 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed0e90efe651499c929b1fb992712b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.571448 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 033c2c016f8f4c01adc8bdbf2885e394 (DEFAULT_TABLE table=default.Table_1ce47a525fa9427198a4b8c589d662c7 [id=c7c35cc13c2840fe969a649388e5d250]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.571894 14428 tablet_service.cc:1505] Processing CreateTablet for tablet ed0e90efe651499c929b1fb992712b96 (DEFAULT_TABLE table=default.Table_eb3b1d93a3ec4891a9d33b131661fd65 [id=46237116be8e46a68f51e20e008f374b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.572435 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 033c2c016f8f4c01adc8bdbf2885e394. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.572728 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed0e90efe651499c929b1fb992712b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.572893 15228 raft_consensus.cc:493] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:39.573256 15228 raft_consensus.cc:515] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.574905 15228 leader_election.cc:290] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:39.574985 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 570c7f08c46e4273bd38c32593dcf304 (DEFAULT_TABLE table=default.Table_f0e893169e89418ab044ae00cdb78176 [id=6ef69beb843946d080b3f6784f0e3af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.575877 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570c7f08c46e4273bd38c32593dcf304. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:39.576575 14230 leader_election.cc:343] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 1976d584cc0a450ab809ead258a251e2
W20251211 08:57:39.576992 14231 leader_election.cc:343] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 1976d584cc0a450ab809ead258a251e2
I20251211 08:57:39.577353 14231 leader_election.cc:304] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:39.578915 15228 raft_consensus.cc:2749] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:39.591981 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 570c7f08c46e4273bd38c32593dcf304 (DEFAULT_TABLE table=default.Table_f0e893169e89418ab044ae00cdb78176 [id=6ef69beb843946d080b3f6784f0e3af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.593091 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570c7f08c46e4273bd38c32593dcf304. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.594252 14578 tablet_service.cc:1505] Processing CreateTablet for tablet c112dfcc301648cda6f0f038b1f2af83 (DEFAULT_TABLE table=default.Table_cfa8f9f640e54badacad03895e72ca57 [id=ffd28388b51247268985b51fdaf33bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.595244 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c112dfcc301648cda6f0f038b1f2af83. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:39.609064 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 47488.8us
I20251211 08:57:39.612725 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 570c7f08c46e4273bd38c32593dcf304 (DEFAULT_TABLE table=default.Table_f0e893169e89418ab044ae00cdb78176 [id=6ef69beb843946d080b3f6784f0e3af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.613730 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570c7f08c46e4273bd38c32593dcf304. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.623222 14269 tablet_service.cc:1505] Processing CreateTablet for tablet ed0e90efe651499c929b1fb992712b96 (DEFAULT_TABLE table=default.Table_eb3b1d93a3ec4891a9d33b131661fd65 [id=46237116be8e46a68f51e20e008f374b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.624285 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed0e90efe651499c929b1fb992712b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.632117 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 0fdc020f21aa4f35880a435d7cfb605d (DEFAULT_TABLE table=default.Table_ccad8285d34949138091df8e2fafd435 [id=c747e05fbb284dd396393d08f7383388]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.632591 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 9ccfcacb5b294a628926d34cd31b137e
I20251211 08:57:39.633221 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdc020f21aa4f35880a435d7cfb605d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.633848 14008 catalog_manager.cc:6724] ReplaceTablet: tablet 9ccfcacb5b294a628926d34cd31b137e deleted and replaced by tablet 1065085c701f48c9ad7206c410ce5985
I20251211 08:57:39.634296 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ccfcacb5b294a628926d34cd31b137e with delete_type TABLET_DATA_DELETED (replaced by tablet 1065085c701f48c9ad7206c410ce5985) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:39.634867 15083 raft_consensus.cc:493] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:39.635232 15083 raft_consensus.cc:515] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.637291 15083 leader_election.cc:290] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:39.637838 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ad600554c0946e0bcb0da6a42be2ffd" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:39.638880 14385 leader_election.cc:343] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.639533 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ad600554c0946e0bcb0da6a42be2ffd" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:39.640955 14382 leader_election.cc:343] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.641268 14382 leader_election.cc:304] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:39.642021 15083 raft_consensus.cc:2749] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:39.643306 14576 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ccfcacb5b294a628926d34cd31b137e with delete_type TABLET_DATA_DELETED (replaced by tablet 1065085c701f48c9ad7206c410ce5985) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:39.648295 14576 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c5315f708c04322919c714dc915a483 with delete_type TABLET_DATA_DELETED (replaced by tablet 69aae3e1d7f34c70b0b48ce63089b482) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:39.648583 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet ad903727f79a4180adf047c92d981c7e with delete_type TABLET_DATA_DELETED (replaced by tablet 9cc3373962574253868ababbbe78e57d) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:39.651764 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet 42720f5f5bf445f2b72e513ad4a6d95f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:39.653271 14011 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 32) took 1078 ms (client timeout 1000 ms). Trace:
I20251211 08:57:39.669106 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0
I20251211 08:57:39.670921 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:39 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:39.672258 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:39 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:39.674227 14277 tablet_service.cc:1505] Processing CreateTablet for tablet c112dfcc301648cda6f0f038b1f2af83 (DEFAULT_TABLE table=default.Table_cfa8f9f640e54badacad03895e72ca57 [id=ffd28388b51247268985b51fdaf33bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.675100 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c112dfcc301648cda6f0f038b1f2af83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.676313 14010 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet b3895c6a5e9f4df8bdca8cb77935274b
I20251211 08:57:39.677459 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 1976d584cc0a450ab809ead258a251e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:39.679658 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:39 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:39.684115 15299 ts_tablet_manager.cc:1929] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:39.684804 14265 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ccfcacb5b294a628926d34cd31b137e with delete_type TABLET_DATA_DELETED (replaced by tablet 1065085c701f48c9ad7206c410ce5985) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:39.708873 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 0fdc020f21aa4f35880a435d7cfb605d (DEFAULT_TABLE table=default.Table_ccad8285d34949138091df8e2fafd435 [id=c747e05fbb284dd396393d08f7383388]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.710675 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet da57b72207414512b9744b283bc869fe
I20251211 08:57:39.715293 14005 catalog_manager.cc:6724] ReplaceTablet: tablet da57b72207414512b9744b283bc869fe deleted and replaced by tablet 27cb4228204c4176b1a348960efd5b26
I20251211 08:57:39.718384 15299 log.cc:1199] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/5daba165262e41af8d3481d3b4929092
I20251211 08:57:39.712306 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet da57b72207414512b9744b283bc869fe with delete_type TABLET_DATA_DELETED (replaced by tablet 27cb4228204c4176b1a348960efd5b26) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:39.716078 14427 tablet_service.cc:1505] Processing CreateTablet for tablet c112dfcc301648cda6f0f038b1f2af83 (DEFAULT_TABLE table=default.Table_cfa8f9f640e54badacad03895e72ca57 [id=ffd28388b51247268985b51fdaf33bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.719777 15299 ts_tablet_manager.cc:1950] T 5daba165262e41af8d3481d3b4929092 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:39.720192 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c112dfcc301648cda6f0f038b1f2af83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.721659 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 219) took 1327 ms. Trace:
I20251211 08:57:39.721917 15299 rpcz_store.cc:276] 1211 08:57:38.394556 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.411849 (+ 17293us) service_pool.cc:225] Handling call
1211 08:57:39.451653 (+1039804us) ts_tablet_manager.cc:1151] Deleting tablet 5daba165262e41af8d3481d3b4929092
1211 08:57:39.451688 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.583137 (+131449us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.683899 (+100762us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.721594 (+ 37695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039330}
I20251211 08:57:39.722944 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 5daba165262e41af8d3481d3b4929092 (table default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]) successfully deleted
I20251211 08:57:39.724797 14722 tablet_bootstrap.cc:492] T 40a1f12553934446bfe44f4ed41abf9a P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:39.725209 14722 ts_tablet_manager.cc:1403] T 40a1f12553934446bfe44f4ed41abf9a P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.438s user 0.014s sys 0.005s
I20251211 08:57:39.728111 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 224) took 1302 ms. Trace:
I20251211 08:57:39.729640 14904 ts_tablet_manager.cc:1929] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:39.728406 15299 rpcz_store.cc:276] 1211 08:57:38.425594 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.434333 (+ 8739us) service_pool.cc:225] Handling call
1211 08:57:39.727819 (+1293486us) ts_tablet_manager.cc:1151] Deleting tablet dca342fd9de241b5884600199f06fbeb
1211 08:57:39.727854 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.728066 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1251995}
I20251211 08:57:39.735957 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 225) took 1259 ms. Trace:
I20251211 08:57:39.705852 14714 tablet_bootstrap.cc:492] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:39.736629 14714 ts_tablet_manager.cc:1403] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.346s user 0.010s sys 0.001s
I20251211 08:57:39.736363 15299 rpcz_store.cc:276] 1211 08:57:38.476937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.484769 (+ 7832us) service_pool.cc:225] Handling call
1211 08:57:39.735555 (+1250786us) ts_tablet_manager.cc:1151] Deleting tablet 8e05628011634149b17d298c54074d00
1211 08:57:39.735603 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.735894 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1250240}
I20251211 08:57:39.738581 15299 tablet_replica.cc:333] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:39.739321 15299 raft_consensus.cc:2243] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:39.739198 14714 raft_consensus.cc:359] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.739799 14714 raft_consensus.cc:385] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.740041 14714 raft_consensus.cc:740] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:39.740125 15299 raft_consensus.cc:2272] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:39.740604 14714 consensus_queue.cc:260] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.741984 15299 ts_tablet_manager.cc:1916] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:39.772249 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet da57b72207414512b9744b283bc869fe with delete_type TABLET_DATA_DELETED (replaced by tablet 27cb4228204c4176b1a348960efd5b26) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:39.772866 15030 tablet_bootstrap.cc:492] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:39.773921 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdc020f21aa4f35880a435d7cfb605d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.773964 15030 ts_tablet_manager.cc:1403] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.501s user 0.007s sys 0.014s
I20251211 08:57:39.774387 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 7c0bdd2d0d9f4221bf13812bb7a97025 (DEFAULT_TABLE table=default.Table_02db0ed6d7e34eb388c5e24bdf50bf74 [id=0472449a728a4c908447bb8f97b3a730]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.775208 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c0bdd2d0d9f4221bf13812bb7a97025. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.776163 15030 raft_consensus.cc:359] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.776731 15030 raft_consensus.cc:385] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.776932 15030 raft_consensus.cc:740] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:39.777513 15030 consensus_queue.cc:260] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.780069 15030 ts_tablet_manager.cc:1434] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251211 08:57:39.732942 14265 tablet_service.cc:1505] Processing CreateTablet for tablet 7c0bdd2d0d9f4221bf13812bb7a97025 (DEFAULT_TABLE table=default.Table_02db0ed6d7e34eb388c5e24bdf50bf74 [id=0472449a728a4c908447bb8f97b3a730]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.734778 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet da57b72207414512b9744b283bc869fe with delete_type TABLET_DATA_DELETED (replaced by tablet 27cb4228204c4176b1a348960efd5b26) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:39.781965 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c0bdd2d0d9f4221bf13812bb7a97025. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.736101 14722 raft_consensus.cc:359] T 40a1f12553934446bfe44f4ed41abf9a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:39.773547 14230 consensus_peers.cc:597] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: caf537c461c04a81b66f268ee98dc64e. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:39.782510 14722 raft_consensus.cc:385] T 40a1f12553934446bfe44f4ed41abf9a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.782835 14722 raft_consensus.cc:740] T 40a1f12553934446bfe44f4ed41abf9a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
W20251211 08:57:39.788349 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet dca342fd9de241b5884600199f06fbeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dca342fd9de241b5884600199f06fbeb already in progress: creating tablet
W20251211 08:57:39.789134 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 8e05628011634149b17d298c54074d00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e05628011634149b17d298c54074d00 already in progress: creating tablet
I20251211 08:57:39.790304 14722 consensus_queue.cc:260] T 40a1f12553934446bfe44f4ed41abf9a P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.797703 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 7c0bdd2d0d9f4221bf13812bb7a97025 (DEFAULT_TABLE table=default.Table_02db0ed6d7e34eb388c5e24bdf50bf74 [id=0472449a728a4c908447bb8f97b3a730]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.799506 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c0bdd2d0d9f4221bf13812bb7a97025. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.799882 14904 log.cc:1199] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/9168007ca0584954a734e2c8f2e73f70
I20251211 08:57:39.804279 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 1b2445f66be54d3fbdc3be964b9a9132 (DEFAULT_TABLE table=default.Table_e71106e05fbc47248f2b5f71482c2231 [id=2cc6cd83f4414e91b8359c3b65336061]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.800199 14722 ts_tablet_manager.cc:1434] T 40a1f12553934446bfe44f4ed41abf9a P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.075s user 0.006s sys 0.000s
I20251211 08:57:39.815706 14722 tablet_bootstrap.cc:492] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:39.819634 14904 ts_tablet_manager.cc:1950] T 9168007ca0584954a734e2c8f2e73f70 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:39.822255 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 207) took 1027 ms. Trace:
I20251211 08:57:39.805580 15030 tablet_bootstrap.cc:492] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:39.829192 14714 ts_tablet_manager.cc:1434] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.092s user 0.005s sys 0.000s
I20251211 08:57:39.830488 14714 tablet_bootstrap.cc:492] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:39.653579 14011 rpcz_store.cc:269] 1211 08:57:38.574357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.574441 (+ 84us) service_pool.cc:225] Handling call
1211 08:57:39.653108 (+1078667us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:39.506795 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:39.507194 (+ 399us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:39.507227 (+ 33us) write_op.cc:435] Acquired schema lock
1211 08:57:39.507249 (+ 22us) tablet.cc:660] Decoding operations
1211 08:57:39.507610 (+ 361us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:39.507663 (+ 53us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:39.507680 (+ 17us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:39.507940 (+ 260us) tablet.cc:699] Row locks acquired
1211 08:57:39.507961 (+ 21us) write_op.cc:260] PREPARE: finished
1211 08:57:39.508062 (+ 101us) write_op.cc:270] Start()
1211 08:57:39.508165 (+ 103us) write_op.cc:276] Timestamp: P: 1765443459508361 usec, L: 0
1211 08:57:39.508197 (+ 32us) op_driver.cc:348] REPLICATION: starting
1211 08:57:39.554594 (+ 46397us) spinlock_profiling.cc:243] Waited 45 ms on lock 0x7b5400020920. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbef5a1 00007fc20bc03550 00007fc20bc8c7c2 00007fc20bc81be7 00007fc20bc81a24 00007fc20c0f34d1 00007fc20c0f2ca8 00007fc20c0f7941 00007fc20c0f78f9 00007fc20c0f7889 00007fc20c0f7851
1211 08:57:39.555234 (+ 640us) log.cc:844] Serialized 907 byte log entry
1211 08:57:39.571842 (+ 16608us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b580002fcd0. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bbe046c 00007fc20bbe6d83 00007fc20bbe6d09 00007fc20bbe6c99 00007fc20bbe6c61 00007fc20bbe5e2d 00000000005aa2b4 00000000005aa0e9 00007fc204bda065 00007fc204bdc3b1
1211 08:57:39.625684 (+ 53842us) op_driver.cc:464] REPLICATION: finished
1211 08:57:39.625907 (+ 223us) write_op.cc:301] APPLY: starting
1211 08:57:39.625972 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:39.626120 (+ 148us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:39.626164 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:39.626577 (+ 413us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:39.626712 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:39.626756 (+ 44us) write_op.cc:312] APPLY: finished
1211 08:57:39.626952 (+ 196us) log.cc:844] Serialized 29 byte log entry
1211 08:57:39.627113 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:39.627173 (+ 60us) write_op.cc:454] Released schema lock
1211 08:57:39.627211 (+ 38us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":79,"apply.run_cpu_time_us":1492,"apply.run_wall_time_us":1498,"num_ops":2,"prepare.queue_time_us":52915,"prepare.run_cpu_time_us":4222,"prepare.run_wall_time_us":48764,"raft.queue_time_us":2265,"raft.run_cpu_time_us":1304,"raft.run_wall_time_us":14164,"replication_time_us":117385,"spinlock_wait_cycles":133595264}]]}
I20251211 08:57:39.832949 15030 tablet_bootstrap.cc:654] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:39.835198 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 78c6a703e1e44f41931fe465def01bf6 (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]). Expected version 2 got 0
I20251211 08:57:39.835844 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2445f66be54d3fbdc3be964b9a9132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.835978 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet e8a8437b8b464fb8ab02a9c552f81db3 (table default.Table_b19d002dfb154df6aadac72b8859cce9 [id=0944cc290d954cf4a33fec11acb53315]). Expected version 1 got 0
I20251211 08:57:39.837347 14004 catalog_manager.cc:5654] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } }
I20251211 08:57:39.838842 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 182) took 2003 ms. Trace:
I20251211 08:57:39.839275 14905 rpcz_store.cc:276] 1211 08:57:37.835070 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:37.835190 (+ 120us) service_pool.cc:225] Handling call
1211 08:57:39.249951 (+1414761us) ts_tablet_manager.cc:1151] Deleting tablet e6c4d4c99b3b496cab125e13153349cb
1211 08:57:39.249993 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.340295 (+ 90302us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.492263 (+151968us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.838787 (+346524us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1408721}
I20251211 08:57:39.841421 14905 tablet_replica.cc:333] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:39.842293 14905 raft_consensus.cc:2243] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:39.844337 14714 tablet_bootstrap.cc:654] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:39.845970 14722 tablet_bootstrap.cc:654] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:39.852921 15193 consensus_queue.cc:1048] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:39.822512 14904 rpcz_store.cc:276] 1211 08:57:38.794582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.794668 (+ 86us) service_pool.cc:225] Handling call
1211 08:57:39.271011 (+476343us) ts_tablet_manager.cc:1151] Deleting tablet 9168007ca0584954a734e2c8f2e73f70
1211 08:57:39.271045 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.334301 (+ 63256us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.729426 (+395125us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.822200 (+ 92774us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":395423}
I20251211 08:57:39.855922 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 208) took 1061 ms. Trace:
I20251211 08:57:39.856207 14904 rpcz_store.cc:276] 1211 08:57:38.794840 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.795061 (+ 221us) service_pool.cc:225] Handling call
1211 08:57:39.855570 (+1060509us) ts_tablet_manager.cc:1151] Deleting tablet e9905227c0a844d199362a442e7d630c
1211 08:57:39.855608 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.855873 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":979799}
I20251211 08:57:39.858446 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 205) took 1065 ms. Trace:
W20251211 08:57:39.861121 14231 consensus_peers.cc:597] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: caf537c461c04a81b66f268ee98dc64e. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:39.866779 14714 tablet_bootstrap.cc:492] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:39.867230 14714 ts_tablet_manager.cc:1403] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.000s
I20251211 08:57:39.869805 14714 raft_consensus.cc:359] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.871294 14714 raft_consensus.cc:385] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.871595 14714 raft_consensus.cc:740] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:39.872951 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 1b2445f66be54d3fbdc3be964b9a9132 (DEFAULT_TABLE table=default.Table_e71106e05fbc47248f2b5f71482c2231 [id=2cc6cd83f4414e91b8359c3b65336061]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.873828 14265 tablet_service.cc:1505] Processing CreateTablet for tablet 1b2445f66be54d3fbdc3be964b9a9132 (DEFAULT_TABLE table=default.Table_e71106e05fbc47248f2b5f71482c2231 [id=2cc6cd83f4414e91b8359c3b65336061]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.875901 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2445f66be54d3fbdc3be964b9a9132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.880286 14714 consensus_queue.cc:260] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.895829 14905 raft_consensus.cc:2272] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:39.888185 15030 tablet_bootstrap.cc:492] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:39.896595 15030 ts_tablet_manager.cc:1403] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.004s
I20251211 08:57:39.899827 15030 raft_consensus.cc:359] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.900364 15030 raft_consensus.cc:385] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.900575 15030 raft_consensus.cc:740] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:39.901245 15030 consensus_queue.cc:260] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.875923 14722 tablet_bootstrap.cc:492] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:39.902683 14722 ts_tablet_manager.cc:1403] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.087s user 0.009s sys 0.002s
I20251211 08:57:39.876147 14904 rpcz_store.cc:276] 1211 08:57:38.793450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.793553 (+ 103us) service_pool.cc:225] Handling call
1211 08:57:39.856921 (+1063368us) ts_tablet_manager.cc:1151] Deleting tablet dca342fd9de241b5884600199f06fbeb
1211 08:57:39.858187 (+ 1266us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.858397 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":980956}
I20251211 08:57:39.904198 15030 ts_tablet_manager.cc:1434] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20251211 08:57:39.904083 14385 consensus_peers.cc:597] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:39.904486 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 206) took 1110 ms. Trace:
I20251211 08:57:39.840863 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet e6c4d4c99b3b496cab125e13153349cb (table default.Table_70f3ebdd01dc4e75bc85ab29528865fc [id=67d1517b43294838bda420b792edb9c5]) successfully deleted
W20251211 08:57:39.905622 13997 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 64809.8us
I20251211 08:57:39.906024 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2445f66be54d3fbdc3be964b9a9132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.873030 14010 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ba237fb232b24588ac35a6455f9476b5" } new_table_name: "default.Table_e83a6dbb039a4f789964450c022037e6" modify_external_catalogs: true
I20251211 08:57:39.874320 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_09b494206172425ba445408538dfebcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:39.911593 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09b494206172425ba445408538dfebcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:39.884867 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 9168007ca0584954a734e2c8f2e73f70 (table default.Table_b997f016aa90456c8c6b1a4dcf5f4c75 [id=dece67928a044390a750633247cfc2cd]) successfully deleted
I20251211 08:57:39.916390 14714 ts_tablet_manager.cc:1434] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
W20251211 08:57:39.916877 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet e9905227c0a844d199362a442e7d630c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9905227c0a844d199362a442e7d630c already in progress: creating tablet
W20251211 08:57:39.918994 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet dca342fd9de241b5884600199f06fbeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dca342fd9de241b5884600199f06fbeb already in progress: creating tablet
I20251211 08:57:39.919425 14714 tablet_bootstrap.cc:492] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:39.920845 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 1c02cb6ec01041dba118683a57aecde2 (DEFAULT_TABLE table=default.Table_874fb7be95234d289bc2b30e873dabe9 [id=612e052e535b4b34ad98f334e8af58ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.921835 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c02cb6ec01041dba118683a57aecde2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.905365 14904 rpcz_store.cc:276] 1211 08:57:38.794291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.803675 (+ 9384us) service_pool.cc:225] Handling call
1211 08:57:39.904199 (+1100524us) ts_tablet_manager.cc:1151] Deleting tablet 8e05628011634149b17d298c54074d00
1211 08:57:39.904239 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.904441 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027140}
I20251211 08:57:39.924496 14714 tablet_bootstrap.cc:654] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:39.885562 14011 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7422c10fa9f240fe82576a2917407bef" } new_table_name: "default.Table_fd2daf1c25534c25b34cf363453b0e67" modify_external_catalogs: true
I20251211 08:57:39.926640 14904 tablet_replica.cc:333] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:39.933638 14904 raft_consensus.cc:2243] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:39.934047 14904 raft_consensus.cc:2272] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:39.935567 14904 ts_tablet_manager.cc:1916] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:39.939028 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 40a1f12553934446bfe44f4ed41abf9a (table default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127]). Expected version 1 got 0
I20251211 08:57:39.940879 14004 catalog_manager.cc:5654] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:39.909317 14722 raft_consensus.cc:359] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.942550 14722 raft_consensus.cc:385] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:39.942976 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 1c02cb6ec01041dba118683a57aecde2 (DEFAULT_TABLE table=default.Table_874fb7be95234d289bc2b30e873dabe9 [id=612e052e535b4b34ad98f334e8af58ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.944360 14722 raft_consensus.cc:740] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:39.944937 14722 consensus_queue.cc:260] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.945324 15152 raft_consensus.cc:493] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:39.945734 15152 raft_consensus.cc:515] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.946676 14905 ts_tablet_manager.cc:1916] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:39.947908 15152 leader_election.cc:290] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:39.950348 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 (DEFAULT_TABLE table=default.Table_70f3ebdd01dc4e75bc85ab29528865fc [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.951282 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c02cb6ec01041dba118683a57aecde2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.952361 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b90b0c944174b2a82696e3fee806934" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:39.955016 14544 leader_election.cc:343] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:39.968724 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 1c02cb6ec01041dba118683a57aecde2 (DEFAULT_TABLE table=default.Table_874fb7be95234d289bc2b30e873dabe9 [id=612e052e535b4b34ad98f334e8af58ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.969982 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c02cb6ec01041dba118683a57aecde2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.972019 14722 ts_tablet_manager.cc:1434] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.069s user 0.005s sys 0.000s
I20251211 08:57:39.972479 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b90b0c944174b2a82696e3fee806934" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:39.972867 14722 tablet_bootstrap.cc:492] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:39.973116 14449 raft_consensus.cc:2468] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:39.973325 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 (DEFAULT_TABLE table=default.Table_70f3ebdd01dc4e75bc85ab29528865fc [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.974215 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d8d84e594ea47f3bc9127cc587e8aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.976504 14542 leader_election.cc:304] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:39.977561 15318 raft_consensus.cc:2804] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:39.977830 15318 raft_consensus.cc:493] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:39.978099 15318 raft_consensus.cc:3060] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:39.978901 15299 ts_tablet_manager.cc:1929] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:39.979336 15299 log.cc:1199] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/9168007ca0584954a734e2c8f2e73f70
I20251211 08:57:39.980152 15299 ts_tablet_manager.cc:1950] T 9168007ca0584954a734e2c8f2e73f70 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:39.980630 14586 tablet_service.cc:1505] Processing CreateTablet for tablet e36c6e2868e143be8e7f72307202a268 (DEFAULT_TABLE table=default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.982136 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 227) took 1338 ms. Trace:
I20251211 08:57:39.982407 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 9cc3373962574253868ababbbe78e57d (DEFAULT_TABLE table=default.Table_dda8864ad51542599828719c4cb52e5b [id=2c2c410df07d4e4cbc60e08dce22efe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:39.983515 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_bd6d3c8d04194e688e953991eb51e39a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:39.984442 14424 tablet_service.cc:1505] Processing CreateTablet for tablet e36c6e2868e143be8e7f72307202a268 (DEFAULT_TABLE table=default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:39.984880 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd6d3c8d04194e688e953991eb51e39a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:39.985464 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36c6e2868e143be8e7f72307202a268. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.987977 15318 raft_consensus.cc:515] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:39.993439 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b90b0c944174b2a82696e3fee806934" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:39.993872 14449 raft_consensus.cc:3060] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:40.001556 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 9cc3373962574253868ababbbe78e57d (DEFAULT_TABLE table=default.Table_dda8864ad51542599828719c4cb52e5b [id=2c2c410df07d4e4cbc60e08dce22efe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.002471 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc3373962574253868ababbbe78e57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.002974 14449 raft_consensus.cc:2468] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:40.004171 14542 leader_election.cc:304] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:40.004909 15152 raft_consensus.cc:2804] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:40.011590 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36c6e2868e143be8e7f72307202a268. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.012128 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc3373962574253868ababbbe78e57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.942270 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet d4f826d079774152a3b66bfb81f990a8 (table default.Table_4ab19ee3f1f3473fa3d320e7841a201c [id=4208ab42111742f38028caeb7913a1c7]). Expected version 1 got 0
W20251211 08:57:40.013574 14010 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e36c6e2868e143be8e7f72307202a268 consensus state has no leader
I20251211 08:57:40.016402 15318 leader_election.cc:290] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:40.020102 15193 raft_consensus.cc:493] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:40.020499 15193 raft_consensus.cc:515] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.022234 15193 leader_election.cc:290] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:40.023118 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d8d84e594ea47f3bc9127cc587e8aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.024181 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41d6ef975a47489385791065bb8f9f46" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:40.025332 14382 leader_election.cc:343] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.028688 14420 tablet_service.cc:1505] Processing CreateTablet for tablet ac6be204692a44fab2db551e79237299 (DEFAULT_TABLE table=default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.029765 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac6be204692a44fab2db551e79237299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.029580 14585 tablet_service.cc:1505] Processing CreateTablet for tablet ac6be204692a44fab2db551e79237299 (DEFAULT_TABLE table=default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.030447 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac6be204692a44fab2db551e79237299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.031467 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_3867d5bf46e94dd1990e38dd47037918"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:40.031559 14422 tablet_service.cc:1505] Processing CreateTablet for tablet 6874fb018bc844b3aeeab7db8fc53e9a (DEFAULT_TABLE table=default.Table_ccb3af786cad4564b032d78520da1cc3 [id=62e4ec0c216b41eebee5490eaba6ef5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.031679 14012 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_358b3598ddbe4cc2831ba06d3600ac09" } modify_external_catalogs: true
I20251211 08:57:40.032191 14012 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_358b3598ddbe4cc2831ba06d3600ac09" } modify_external_catalogs: true
I20251211 08:57:40.032516 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6874fb018bc844b3aeeab7db8fc53e9a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:40.032750 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3867d5bf46e94dd1990e38dd47037918 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:40.044312 15030 tablet_bootstrap.cc:492] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:40.049659 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 6874fb018bc844b3aeeab7db8fc53e9a (DEFAULT_TABLE table=default.Table_ccb3af786cad4564b032d78520da1cc3 [id=62e4ec0c216b41eebee5490eaba6ef5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.051128 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6874fb018bc844b3aeeab7db8fc53e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.053973 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41d6ef975a47489385791065bb8f9f46" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:40.055685 14385 leader_election.cc:343] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.056012 14385 leader_election.cc:304] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:40.056281 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b90b0c944174b2a82696e3fee806934" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:40.057231 15083 raft_consensus.cc:2749] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:40.058607 14544 leader_election.cc:343] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.053148 15152 raft_consensus.cc:697] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:40.060335 15152 consensus_queue.cc:237] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.062405 15193 consensus_queue.cc:1048] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:40.081677 14006 catalog_manager.cc:5654] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:40.083407 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_73091d56834e4e78b870257988c87c57" } modify_external_catalogs: true
I20251211 08:57:40.083966 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_73091d56834e4e78b870257988c87c57" } modify_external_catalogs: true
I20251211 08:57:40.084056 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_765e4ba63db247d4b7c358923033b6b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:40.085353 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_765e4ba63db247d4b7c358923033b6b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:40.082991 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet e8a8437b8b464fb8ab02a9c552f81db3 (table default.Table_b19d002dfb154df6aadac72b8859cce9 [id=0944cc290d954cf4a33fec11acb53315]). Expected version 1 got 0
I20251211 08:57:40.087443 14006 catalog_manager.cc:5654] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:40.088606 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 78c6a703e1e44f41931fe465def01bf6 (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]). Expected version 2 got 0
I20251211 08:57:40.089303 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 40a1f12553934446bfe44f4ed41abf9a (table default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127]). Expected version 1 got 0
I20251211 08:57:40.090153 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet d4f826d079774152a3b66bfb81f990a8 (table default.Table_4ab19ee3f1f3473fa3d320e7841a201c [id=4208ab42111742f38028caeb7913a1c7]). Expected version 1 got 0
I20251211 08:57:40.101327 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 (DEFAULT_TABLE table=default.Table_70f3ebdd01dc4e75bc85ab29528865fc [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.101514 14275 tablet_service.cc:1505] Processing CreateTablet for tablet ac6be204692a44fab2db551e79237299 (DEFAULT_TABLE table=default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.102675 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d8d84e594ea47f3bc9127cc587e8aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.103452 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac6be204692a44fab2db551e79237299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.107764 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 6874fb018bc844b3aeeab7db8fc53e9a (DEFAULT_TABLE table=default.Table_ccb3af786cad4564b032d78520da1cc3 [id=62e4ec0c216b41eebee5490eaba6ef5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.109009 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6874fb018bc844b3aeeab7db8fc53e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:39.982432 15299 rpcz_store.cc:276] 1211 08:57:38.644070 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.652860 (+ 8790us) service_pool.cc:225] Handling call
1211 08:57:39.738452 (+1085592us) ts_tablet_manager.cc:1151] Deleting tablet 9168007ca0584954a734e2c8f2e73f70
1211 08:57:39.738488 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:39.868280 (+129792us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.933907 (+ 65627us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.982070 (+ 48163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":850225}
I20251211 08:57:40.113873 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 476eee1606e74832abf928dbaafc844d (DEFAULT_TABLE table=default.Table_05c727e9a723434ba4239e838c6afc9e [id=74ec01634f3a4dee9a90892189ff6462]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.117522 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 228) took 1433 ms. Trace:
I20251211 08:57:40.119084 15299 rpcz_store.cc:276] 1211 08:57:38.683706 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.687364 (+ 3658us) service_pool.cc:225] Handling call
1211 08:57:40.117164 (+1429800us) ts_tablet_manager.cc:1151] Deleting tablet e9905227c0a844d199362a442e7d630c
1211 08:57:40.117229 (+ 65us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.117476 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1226740}
I20251211 08:57:40.123044 15360 consensus_queue.cc:1048] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:40.124729 14714 tablet_bootstrap.cc:492] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:40.125262 14714 ts_tablet_manager.cc:1403] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.206s user 0.009s sys 0.000s
I20251211 08:57:40.116670 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 9168007ca0584954a734e2c8f2e73f70 (table default.Table_b997f016aa90456c8c6b1a4dcf5f4c75 [id=dece67928a044390a750633247cfc2cd]) successfully deleted
W20251211 08:57:40.129081 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet e9905227c0a844d199362a442e7d630c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9905227c0a844d199362a442e7d630c already in progress: creating tablet
I20251211 08:57:40.134965 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 4e813a4c73334c839bfe6af2119e259c
I20251211 08:57:40.139823 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 229) took 1071 ms. Trace:
I20251211 08:57:40.140290 15299 rpcz_store.cc:276] 1211 08:57:39.068234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.068334 (+ 100us) service_pool.cc:225] Handling call
1211 08:57:40.120764 (+1052430us) ts_tablet_manager.cc:1151] Deleting tablet f785a3cc2e8145eebae7e9c2885b0b1e
1211 08:57:40.120855 (+ 91us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.139778 (+ 18923us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051848}
W20251211 08:57:40.141300 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f785a3cc2e8145eebae7e9c2885b0b1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f785a3cc2e8145eebae7e9c2885b0b1e already in progress: creating tablet
I20251211 08:57:40.142274 14714 raft_consensus.cc:359] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.142881 14714 raft_consensus.cc:385] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:40.143102 14714 raft_consensus.cc:740] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:40.144702 14714 consensus_queue.cc:260] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.146795 14714 ts_tablet_manager.cc:1434] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.021s user 0.003s sys 0.004s
I20251211 08:57:40.147768 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e813a4c73334c839bfe6af2119e259c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:40.151240 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 8e05628011634149b17d298c54074d00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e05628011634149b17d298c54074d00 already in progress: creating tablet
I20251211 08:57:40.153592 14269 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e813a4c73334c839bfe6af2119e259c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:40.158934 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e813a4c73334c839bfe6af2119e259c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:40.159457 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 476eee1606e74832abf928dbaafc844d (DEFAULT_TABLE table=default.Table_05c727e9a723434ba4239e838c6afc9e [id=74ec01634f3a4dee9a90892189ff6462]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:40.159821 14385 consensus_peers.cc:597] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:40.160501 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476eee1606e74832abf928dbaafc844d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.160578 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 230) took 1084 ms. Trace:
I20251211 08:57:40.160861 15299 rpcz_store.cc:276] 1211 08:57:39.075942 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.076040 (+ 98us) service_pool.cc:225] Handling call
1211 08:57:40.141377 (+1065337us) ts_tablet_manager.cc:1151] Deleting tablet f3403116517f45c59f87fa5bda0b4071
1211 08:57:40.160253 (+ 18876us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.160530 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1064693}
W20251211 08:57:40.161792 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f3403116517f45c59f87fa5bda0b4071 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3403116517f45c59f87fa5bda0b4071 already in progress: creating tablet
I20251211 08:57:40.162066 15299 tablet_replica.cc:333] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:40.163498 14714 tablet_bootstrap.cc:492] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:40.163462 14266 tablet_service.cc:1505] Processing CreateTablet for tablet e36c6e2868e143be8e7f72307202a268 (DEFAULT_TABLE table=default.Table_ba237fb232b24588ac35a6455f9476b5 [id=8f146b956cf348b49a4d9694bfaca344]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.164412 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36c6e2868e143be8e7f72307202a268. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.167423 14268 tablet_service.cc:1505] Processing CreateTablet for tablet 9cc3373962574253868ababbbe78e57d (DEFAULT_TABLE table=default.Table_dda8864ad51542599828719c4cb52e5b [id=2c2c410df07d4e4cbc60e08dce22efe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.168305 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc3373962574253868ababbbe78e57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.168437 14714 tablet_bootstrap.cc:654] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:40.171415 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476eee1606e74832abf928dbaafc844d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.177883 14426 tablet_service.cc:1505] Processing CreateTablet for tablet d7a8acb51a2948f4a018787bfaecb694 (DEFAULT_TABLE table=default.Table_58973fbd22514d66804dc101b76a7451 [id=de3a242b7e6d480ebf509e458ec002c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.179447 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7a8acb51a2948f4a018787bfaecb694. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:40.191356 14544 consensus_peers.cc:597] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:40.201699 14542 consensus_peers.cc:597] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:40.202230 15299 raft_consensus.cc:2243] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251211 08:57:40.205395 14544 consensus_peers.cc:597] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:40.206367 15299 pending_rounds.cc:70] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:40.209803 14584 tablet_service.cc:1505] Processing CreateTablet for tablet d7a8acb51a2948f4a018787bfaecb694 (DEFAULT_TABLE table=default.Table_58973fbd22514d66804dc101b76a7451 [id=de3a242b7e6d480ebf509e458ec002c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.211532 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7a8acb51a2948f4a018787bfaecb694. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.211658 14722 tablet_bootstrap.cc:654] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:40.138118 14904 ts_tablet_manager.cc:1929] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:40.219400 14904 log.cc:1199] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/a4e307e4d1994af0b99a88b3b06b73a4
I20251211 08:57:40.220144 14904 ts_tablet_manager.cc:1950] T a4e307e4d1994af0b99a88b3b06b73a4 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:40.227162 15299 pending_rounds.cc:77] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256407879163904 op_type: NO_OP noop_request { }
I20251211 08:57:40.231181 15299 raft_consensus.cc:2889] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:40.207257 14449 raft_consensus.cc:1275] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Refusing update from remote peer c5e624bcfd154cd586ef1dedb9940063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:40.233798 14722 tablet_bootstrap.cc:492] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:40.234158 14722 ts_tablet_manager.cc:1403] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.261s user 0.007s sys 0.002s
I20251211 08:57:40.234169 15299 raft_consensus.cc:2272] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:40.236629 15299 ts_tablet_manager.cc:1916] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:40.236349 14722 raft_consensus.cc:359] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.236864 14722 raft_consensus.cc:385] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:40.237102 14722 raft_consensus.cc:740] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:40.244095 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 476eee1606e74832abf928dbaafc844d (DEFAULT_TABLE table=default.Table_05c727e9a723434ba4239e838c6afc9e [id=74ec01634f3a4dee9a90892189ff6462]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.245677 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476eee1606e74832abf928dbaafc844d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.247634 14722 consensus_queue.cc:260] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.250380 15193 raft_consensus.cc:493] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:40.251119 15193 raft_consensus.cc:515] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.253571 14722 ts_tablet_manager.cc:1434] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20251211 08:57:40.254319 14722 tablet_bootstrap.cc:492] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:40.254976 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6570281e1064c79b6e2297723f28658" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:40.256204 14385 leader_election.cc:343] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.265954 15193 leader_election.cc:290] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:40.268348 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 213) took 1086 ms. Trace:
I20251211 08:57:40.268452 14714 tablet_bootstrap.cc:492] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:40.268807 14714 ts_tablet_manager.cc:1403] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.000s
I20251211 08:57:40.268651 14904 rpcz_store.cc:276] 1211 08:57:39.182093 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.182169 (+ 76us) service_pool.cc:225] Handling call
1211 08:57:39.926508 (+744339us) ts_tablet_manager.cc:1151] Deleting tablet a4e307e4d1994af0b99a88b3b06b73a4
1211 08:57:39.926547 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.034731 (+108184us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.137906 (+103175us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.268305 (+130399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":714715}
I20251211 08:57:40.269873 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 212) took 1088 ms. Trace:
I20251211 08:57:40.270183 14904 rpcz_store.cc:276] 1211 08:57:39.181897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.184562 (+ 2665us) service_pool.cc:225] Handling call
1211 08:57:40.269582 (+1085020us) ts_tablet_manager.cc:1151] Deleting tablet f3403116517f45c59f87fa5bda0b4071
1211 08:57:40.269615 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.269813 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1043299}
I20251211 08:57:40.271139 14714 raft_consensus.cc:359] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.271715 14714 raft_consensus.cc:385] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:40.271973 14714 raft_consensus.cc:740] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:40.278257 14267 tablet_service.cc:1505] Processing CreateTablet for tablet d7a8acb51a2948f4a018787bfaecb694 (DEFAULT_TABLE table=default.Table_58973fbd22514d66804dc101b76a7451 [id=de3a242b7e6d480ebf509e458ec002c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.279171 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7a8acb51a2948f4a018787bfaecb694. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.280457 14422 tablet_service.cc:1505] Processing CreateTablet for tablet 07f983604bfa474f9c9ce0c982d738fc (DEFAULT_TABLE table=default.Table_80758110ce794260af2a551849bf4896 [id=277e1a7e34f54c16b29a738c17f75da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.290704 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet a4e307e4d1994af0b99a88b3b06b73a4 (table default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]) successfully deleted
I20251211 08:57:40.254644 15360 consensus_queue.cc:1048] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251211 08:57:40.260181 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 07f983604bfa474f9c9ce0c982d738fc (DEFAULT_TABLE table=default.Table_80758110ce794260af2a551849bf4896 [id=277e1a7e34f54c16b29a738c17f75da5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:40.291298 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet f3403116517f45c59f87fa5bda0b4071 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3403116517f45c59f87fa5bda0b4071 already in progress: creating tablet
I20251211 08:57:40.291894 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f983604bfa474f9c9ce0c982d738fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.292105 14012 master_service.cc:954] ReplaceTablet: received request to replace tablet f5b462cf21e3420c8e323a4ac4f04abe from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:40.295042 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 07f983604bfa474f9c9ce0c982d738fc (DEFAULT_TABLE table=default.Table_80758110ce794260af2a551849bf4896 [id=277e1a7e34f54c16b29a738c17f75da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:40.296466 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f983604bfa474f9c9ce0c982d738fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.298651 14904 tablet_replica.cc:333] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:40.299392 14904 raft_consensus.cc:2243] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:40.300118 14904 pending_rounds.cc:70] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
W20251211 08:57:40.261778 14544 consensus_peers.cc:597] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:40.304028 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6570281e1064c79b6e2297723f28658" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:40.305117 14382 leader_election.cc:343] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.305519 14382 leader_election.cc:304] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:40.308125 15193 raft_consensus.cc:2749] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:40.305325 14714 consensus_queue.cc:260] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.275086 15030 tablet_bootstrap.cc:654] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:40.322216 14714 ts_tablet_manager.cc:1434] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.053s user 0.009s sys 0.000s
I20251211 08:57:40.336710 14905 ts_tablet_manager.cc:1929] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:40.337057 14905 log.cc:1199] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/afe339c4cc95490683618e18a3da252b
I20251211 08:57:40.337702 14905 ts_tablet_manager.cc:1950] T afe339c4cc95490683618e18a3da252b P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:40.344293 14904 pending_rounds.cc:77] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256407900155904 op_type: NO_OP noop_request { }
I20251211 08:57:40.345592 14904 raft_consensus.cc:2889] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:40.346001 14904 raft_consensus.cc:2272] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:40.354272 14542 consensus_peers.cc:597] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daba165262e41af8d3481d3b4929092. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:40.372062 14904 ts_tablet_manager.cc:1916] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:40.352499 14722 tablet_bootstrap.cc:654] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:40.386372 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f983604bfa474f9c9ce0c982d738fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:40.399101 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_c2b453de9d7c46ed92713e510df27dca" } new_table_name: "default.Table_0ff1ebe3d6b144cc94137467d38b30bb" modify_external_catalogs: true
I20251211 08:57:40.399322 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_eed8ebdeeca94232a29625f6435c6ea0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:40.400102 15368 raft_consensus.cc:493] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:40.400485 15368 raft_consensus.cc:515] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:40.400821 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eed8ebdeeca94232a29625f6435c6ea0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:40.402169 15368 leader_election.cc:290] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:40.402896 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_cbfedb62a1204c96adec00226098c500"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:40.404167 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbfedb62a1204c96adec00226098c500 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:40.405135 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c5315f708c04322919c714dc915a483" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:40.405840 14607 raft_consensus.cc:2468] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:40.407655 14230 leader_election.cc:304] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
W20251211 08:57:40.408268 14231 leader_election.cc:343] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 3c5315f708c04322919c714dc915a483
I20251211 08:57:40.409209 15363 raft_consensus.cc:2804] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:40.409569 15363 raft_consensus.cc:493] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:40.409919 15363 raft_consensus.cc:3060] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:40.410403 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_e83a6dbb039a4f789964450c022037e6 [id=8f146b956cf348b49a4d9694bfaca344], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e36c6e2868e143be8e7f72307202a268 consensus state has no leader
I20251211 08:57:40.427860 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 5daba165262e41af8d3481d3b4929092 with delete_type TABLET_DATA_DELETED (replaced by tablet da52cace2aab4f9dacef18ec5fa56aaf) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:40.428578 15376 tablet.cc:1722] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:40.436789 15363 raft_consensus.cc:515] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:40.440497 14231 leader_election.cc:343] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 3c5315f708c04322919c714dc915a483
I20251211 08:57:40.441474 15377 tablet.cc:1722] T d4f826d079774152a3b66bfb81f990a8 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:40.445729 14722 tablet_bootstrap.cc:492] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:40.446197 14722 ts_tablet_manager.cc:1403] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.192s user 0.009s sys 0.000s
I20251211 08:57:40.448355 14722 raft_consensus.cc:359] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.449052 14722 raft_consensus.cc:385] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:40.449348 14722 raft_consensus.cc:740] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:40.452656 14722 consensus_queue.cc:260] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.454320 14298 raft_consensus.cc:3060] T 40a1f12553934446bfe44f4ed41abf9a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:40.487854 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 183) took 2647 ms. Trace:
I20251211 08:57:40.488406 14905 rpcz_store.cc:276] 1211 08:57:37.840676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:37.840805 (+ 129us) service_pool.cc:225] Handling call
1211 08:57:39.841273 (+2000468us) ts_tablet_manager.cc:1151] Deleting tablet afe339c4cc95490683618e18a3da252b
1211 08:57:39.841314 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.273369 (+432055us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.336512 (+ 63143us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.487787 (+151275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1999969}
I20251211 08:57:40.489957 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet afe339c4cc95490683618e18a3da252b (table default.Table_e83a6dbb039a4f789964450c022037e6 [id=8f146b956cf348b49a4d9694bfaca344]) successfully deleted
I20251211 08:57:40.491206 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 185) took 2637 ms. Trace:
I20251211 08:57:40.492688 14722 ts_tablet_manager.cc:1434] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.046s user 0.004s sys 0.000s
I20251211 08:57:40.496196 15363 leader_election.cc:290] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:40.501381 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet a6570281e1064c79b6e2297723f28658 with delete_type TABLET_DATA_DELETED (replaced by tablet e9905227c0a844d199362a442e7d630c) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:40.501958 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ad600554c0946e0bcb0da6a42be2ffd with delete_type TABLET_DATA_DELETED (replaced by tablet 6874fb018bc844b3aeeab7db8fc53e9a) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:40.502619 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 174fefd70bcc41418a77a5716f97cd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:40.427806 14004 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 78c6a703e1e44f41931fe465def01bf6 (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 78c6a703e1e44f41931fe465def01bf6 consensus state has no leader
I20251211 08:57:40.503530 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 9168007ca0584954a734e2c8f2e73f70 with delete_type TABLET_DATA_DELETED (replaced by tablet f56fd93aeac6435ea84388ce05580113) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:40.504642 14004 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 33) took 892 ms (client timeout 1000 ms). Trace:
I20251211 08:57:40.505020 14722 tablet_bootstrap.cc:492] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:40.497112 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c5315f708c04322919c714dc915a483" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:40.510572 14605 raft_consensus.cc:3060] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:40.504319 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet afe339c4cc95490683618e18a3da252b with delete_type TABLET_DATA_DELETED (replaced by tablet e36c6e2868e143be8e7f72307202a268) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:40.516923 14905 rpcz_store.cc:276] 1211 08:57:37.853328 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:37.853955 (+ 627us) service_pool.cc:225] Handling call
1211 08:57:40.490897 (+2636942us) ts_tablet_manager.cc:1151] Deleting tablet c341b762afd8440bb7198ca134c8dffe
1211 08:57:40.490939 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.491164 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2636324}
W20251211 08:57:40.517940 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet c341b762afd8440bb7198ca134c8dffe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c341b762afd8440bb7198ca134c8dffe already in progress: creating tablet
I20251211 08:57:40.518663 14605 raft_consensus.cc:2468] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:40.519699 14905 tablet_replica.cc:333] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:40.520525 14714 tablet_bootstrap.cc:492] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:40.526650 14714 tablet_bootstrap.cc:654] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:40.508540 15030 tablet_bootstrap.cc:492] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:40.527797 15030 ts_tablet_manager.cc:1403] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.623s user 0.018s sys 0.004s
I20251211 08:57:40.529930 15030 raft_consensus.cc:359] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.530481 15030 raft_consensus.cc:385] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:40.530383 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2c5e99b90f0c4c4a8cec03d4471f3364"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:40.531795 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c5e99b90f0c4c4a8cec03d4471f3364 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:40.532933 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 49015c5d4f2b4d07b547a9df57c992eb
I20251211 08:57:40.534718 14005 master_service.cc:954] ReplaceTablet: received request to replace tablet a6570281e1064c79b6e2297723f28658 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:40.535166 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet 49015c5d4f2b4d07b547a9df57c992eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:40.535306 14006 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 31) took 972 ms (client timeout 1000 ms). Trace:
I20251211 08:57:40.534302 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet 49015c5d4f2b4d07b547a9df57c992eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:40.535924 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_874fb7be95234d289bc2b30e873dabe9" } modify_external_catalogs: true
I20251211 08:57:40.535509 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 49015c5d4f2b4d07b547a9df57c992eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:40.536383 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_874fb7be95234d289bc2b30e873dabe9" } modify_external_catalogs: true
I20251211 08:57:40.537801 15384 tablet.cc:1722] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:40.538679 14230 leader_election.cc:304] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:40.539673 15363 raft_consensus.cc:2804] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:40.541074 15299 ts_tablet_manager.cc:1929] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:40.541373 15299 log.cc:1199] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/42720f5f5bf445f2b72e513ad4a6d95f
I20251211 08:57:40.539660 15385 tablet.cc:1722] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:40.541960 15299 ts_tablet_manager.cc:1950] T 42720f5f5bf445f2b72e513ad4a6d95f P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:40.542338 15363 raft_consensus.cc:697] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:40.543659 15363 consensus_queue.cc:237] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.548620 14905 raft_consensus.cc:2243] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:40.549156 14905 raft_consensus.cc:2272] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:40.554282 14904 ts_tablet_manager.cc:1929] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:40.556252 14905 ts_tablet_manager.cc:1916] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:40.542716 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 237) took 1072 ms. Trace:
I20251211 08:57:40.556947 15299 rpcz_store.cc:276] 1211 08:57:39.470206 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.470303 (+ 97us) service_pool.cc:225] Handling call
1211 08:57:40.161916 (+691613us) ts_tablet_manager.cc:1151] Deleting tablet 42720f5f5bf445f2b72e513ad4a6d95f
1211 08:57:40.161962 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.205988 (+ 44026us) spinlock_profiling.cc:243] Waited 39.4 ms on lock 0x7b5400092ed4. stack: 00007efeecdf4083 00007efeecdf3f80 00007efeec969574 0000000000508744 00000000005086d9 00007efef3e3b5a1 00007efef3ed9fe2 00007efef42fc2ec 00007efef4f5331a 00007efef4f83f80 00007efef4f6a54d 00007efef4f6a4f9 00007efef4f6a489 00007efef4f6a451 00007efef4f6967d 00000000005aa2b4
1211 08:57:40.353814 (+147826us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.540863 (+187049us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.542674 (+ 1811us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":90729344,"tablet-delete.queue_time_us":691151}
I20251211 08:57:40.561416 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 42720f5f5bf445f2b72e513ad4a6d95f (table default.Table_58593e18b9634b2c8912ff1109e05a52 [id=8bb6612dda374661a7d3a8a06bd230ae]) successfully deleted
I20251211 08:57:40.561575 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 238) took 1091 ms. Trace:
I20251211 08:57:40.561796 15299 rpcz_store.cc:276] 1211 08:57:39.470485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.470824 (+ 339us) service_pool.cc:225] Handling call
1211 08:57:40.561304 (+1090480us) ts_tablet_manager.cc:1151] Deleting tablet afe339c4cc95490683618e18a3da252b
1211 08:57:40.561350 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.561522 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090032}
W20251211 08:57:40.562973 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet afe339c4cc95490683618e18a3da252b because the tablet was not found. No further retry: Not found: Tablet not found: afe339c4cc95490683618e18a3da252b
I20251211 08:57:40.562741 15299 tablet_replica.cc:333] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:40.567634 14010 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 40a1f12553934446bfe44f4ed41abf9a (table default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127]). Expected version 1 got 0
I20251211 08:57:40.568399 15299 raft_consensus.cc:2243] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:40.568823 15299 raft_consensus.cc:2272] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:40.574906 15383 tablet.cc:1722] T 40a1f12553934446bfe44f4ed41abf9a P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:40.593645 14714 tablet_bootstrap.cc:492] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:40.594103 14714 ts_tablet_manager.cc:1403] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.074s user 0.015s sys 0.002s
I20251211 08:57:40.596207 14714 raft_consensus.cc:359] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.530681 15030 raft_consensus.cc:740] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:40.603222 15030 consensus_queue.cc:260] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:40.604427 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f7775994f684326b6312c802f6b084d consensus state has no leader
I20251211 08:57:40.605095 15030 ts_tablet_manager.cc:1434] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.077s user 0.004s sys 0.000s
I20251211 08:57:40.594089 15390 tablet.cc:1722] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:40.598807 15389 tablet.cc:1722] T 40a1f12553934446bfe44f4ed41abf9a P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:40.614593 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet f5b462cf21e3420c8e323a4ac4f04abe
I20251211 08:57:40.615955 14012 catalog_manager.cc:6724] ReplaceTablet: tablet f5b462cf21e3420c8e323a4ac4f04abe deleted and replaced by tablet 2ed430a2c82d498a9c3ca116eaac154e
I20251211 08:57:40.616796 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b462cf21e3420c8e323a4ac4f04abe with delete_type TABLET_DATA_DELETED (replaced by tablet 2ed430a2c82d498a9c3ca116eaac154e) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:40.535571 14006 rpcz_store.cc:269] 1211 08:57:39.563092 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.563231 (+ 139us) service_pool.cc:225] Handling call
1211 08:57:39.597062 (+ 33831us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:40.535259 (+938197us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:40.412765 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:40.413113 (+ 348us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:40.413135 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:57:40.413146 (+ 11us) tablet.cc:660] Decoding operations
1211 08:57:40.413410 (+ 264us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:40.413458 (+ 48us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:40.413479 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:40.413727 (+ 248us) tablet.cc:699] Row locks acquired
1211 08:57:40.413742 (+ 15us) write_op.cc:260] PREPARE: finished
1211 08:57:40.413869 (+ 127us) write_op.cc:270] Start()
1211 08:57:40.413942 (+ 73us) write_op.cc:276] Timestamp: P: 1765443460414008 usec, L: 0
1211 08:57:40.413955 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:40.414351 (+ 396us) log.cc:844] Serialized 907 byte log entry
1211 08:57:40.437727 (+ 23376us) op_driver.cc:464] REPLICATION: finished
1211 08:57:40.442595 (+ 4868us) write_op.cc:301] APPLY: starting
1211 08:57:40.442670 (+ 75us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:40.442893 (+ 223us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:40.443054 (+ 161us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:40.443536 (+ 482us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:40.443735 (+ 199us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:40.443849 (+ 114us) write_op.cc:312] APPLY: finished
1211 08:57:40.444146 (+ 297us) log.cc:844] Serialized 29 byte log entry
1211 08:57:40.444464 (+ 318us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:40.444559 (+ 95us) write_op.cc:454] Released schema lock
1211 08:57:40.444619 (+ 60us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:40.446841 (+ 2222us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:40.446915 (+ 74us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":318142,"reported_tablets":15,"spinlock_wait_cycles":67534208,"child_traces":[["op",{"apply.queue_time_us":4603,"apply.run_cpu_time_us":1729,"apply.run_wall_time_us":4121,"num_ops":2,"prepare.queue_time_us":131,"prepare.run_cpu_time_us":1857,"prepare.run_wall_time_us":1857,"raft.queue_time_us":919,"raft.run_cpu_time_us":526,"raft.run_wall_time_us":526,"replication_time_us":23694}]]}
W20251211 08:57:40.504920 14004 rpcz_store.cc:269] 1211 08:57:39.611931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.612032 (+ 101us) service_pool.cc:225] Handling call
1211 08:57:40.504592 (+892560us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:40.014913 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:40.015245 (+ 332us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:40.015269 (+ 24us) write_op.cc:435] Acquired schema lock
1211 08:57:40.015279 (+ 10us) tablet.cc:660] Decoding operations
1211 08:57:40.015572 (+ 293us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:40.015607 (+ 35us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:40.015628 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:40.015860 (+ 232us) tablet.cc:699] Row locks acquired
1211 08:57:40.015873 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:40.015966 (+ 93us) write_op.cc:270] Start()
1211 08:57:40.016057 (+ 91us) write_op.cc:276] Timestamp: P: 1765443460015945 usec, L: 0
1211 08:57:40.016071 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:40.016527 (+ 456us) log.cc:844] Serialized 907 byte log entry
1211 08:57:40.067183 (+ 50656us) op_driver.cc:464] REPLICATION: finished
1211 08:57:40.067537 (+ 354us) write_op.cc:301] APPLY: starting
1211 08:57:40.067611 (+ 74us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:40.067716 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:40.067804 (+ 88us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:40.068107 (+ 303us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:40.068261 (+ 154us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:40.068301 (+ 40us) write_op.cc:312] APPLY: finished
1211 08:57:40.068479 (+ 178us) log.cc:844] Serialized 29 byte log entry
1211 08:57:40.068645 (+ 166us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:40.068709 (+ 64us) write_op.cc:454] Released schema lock
1211 08:57:40.068741 (+ 32us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:40.068970 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:40.068995 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":224,"apply.run_cpu_time_us":1387,"apply.run_wall_time_us":1385,"num_ops":2,"prepare.queue_time_us":345,"prepare.run_cpu_time_us":1862,"prepare.run_wall_time_us":1861,"raft.queue_time_us":167,"raft.run_cpu_time_us":503,"raft.run_wall_time_us":502,"replication_time_us":51018}]]}
I20251211 08:57:40.644248 15030 tablet_bootstrap.cc:492] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:40.659924 15318 raft_consensus.cc:493] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:40.660327 15318 raft_consensus.cc:515] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.662060 15318 leader_election.cc:290] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:40.669296 15390 tablet.cc:1722] T e8a8437b8b464fb8ab02a9c552f81db3 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:40.611146 15299 ts_tablet_manager.cc:1916] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:40.620661 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b462cf21e3420c8e323a4ac4f04abe with delete_type TABLET_DATA_DELETED (replaced by tablet 2ed430a2c82d498a9c3ca116eaac154e) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:40.659268 15391 tablet.cc:1722] T e8a8437b8b464fb8ab02a9c552f81db3 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:40.664124 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db302282bc3d4d8cb56d1cbb2f73c4fe" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:40.680403 14542 leader_election.cc:343] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.607967 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c5315f708c04322919c714dc915a483 with delete_type TABLET_DATA_DELETED (replaced by tablet 69aae3e1d7f34c70b0b48ce63089b482) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:40.676728 15193 raft_consensus.cc:493] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:40.619441 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b462cf21e3420c8e323a4ac4f04abe with delete_type TABLET_DATA_DELETED (replaced by tablet 2ed430a2c82d498a9c3ca116eaac154e) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:40.690930 14904 log.cc:1199] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/caf537c461c04a81b66f268ee98dc64e
I20251211 08:57:40.691551 14904 ts_tablet_manager.cc:1950] T caf537c461c04a81b66f268ee98dc64e P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:40.690899 15193 raft_consensus.cc:515] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.692247 14905 ts_tablet_manager.cc:1929] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:40.640228 15386 tablet.cc:1722] T 40a1f12553934446bfe44f4ed41abf9a P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:40.692600 14905 log.cc:1199] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/81cfb3e0baef45ddab4a77a4e0790085
I20251211 08:57:40.693286 14905 ts_tablet_manager.cc:1950] T 81cfb3e0baef45ddab4a77a4e0790085 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:40.695062 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 187) took 2655 ms. Trace:
I20251211 08:57:40.695459 14905 rpcz_store.cc:276] 1211 08:57:38.039688 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.065996 (+ 26308us) service_pool.cc:225] Handling call
1211 08:57:40.519567 (+2453571us) ts_tablet_manager.cc:1151] Deleting tablet 81cfb3e0baef45ddab4a77a4e0790085
1211 08:57:40.519612 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.633648 (+114036us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.692056 (+ 58408us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.695013 (+ 2957us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2452998}
I20251211 08:57:40.696463 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 81cfb3e0baef45ddab4a77a4e0790085 (table default.Table_02e79cde30df4a2d8e81657161c2a56e [id=8ac1575fb83e405493d250160ca6ac24]) successfully deleted
I20251211 08:57:40.697413 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 190) took 2593 ms. Trace:
I20251211 08:57:40.697660 14905 rpcz_store.cc:276] 1211 08:57:38.103620 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.103743 (+ 123us) service_pool.cc:225] Handling call
1211 08:57:40.697071 (+2593328us) ts_tablet_manager.cc:1151] Deleting tablet 4eec7694087145d8a49dd93904f95980
1211 08:57:40.697103 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.697373 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2592624}
W20251211 08:57:40.698719 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 4eec7694087145d8a49dd93904f95980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4eec7694087145d8a49dd93904f95980 already in progress: creating tablet
I20251211 08:57:40.642469 14722 tablet_bootstrap.cc:654] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:40.689277 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db302282bc3d4d8cb56d1cbb2f73c4fe" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:40.699635 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 194) took 2360 ms. Trace:
I20251211 08:57:40.699957 14905 rpcz_store.cc:276] 1211 08:57:38.339143 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.371545 (+ 32402us) service_pool.cc:225] Handling call
1211 08:57:40.699415 (+2327870us) ts_tablet_manager.cc:1151] Deleting tablet 2cd6e4352cfc41e497dc58d158e24ff2
1211 08:57:40.699451 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.699582 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2324997}
W20251211 08:57:40.700481 14544 leader_election.cc:343] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:40.700932 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 2cd6e4352cfc41e497dc58d158e24ff2 because the tablet was not found. No further retry: Not found: Tablet not found: 2cd6e4352cfc41e497dc58d158e24ff2
I20251211 08:57:40.701016 14544 leader_election.cc:304] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:40.701865 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 204) took 2170 ms. Trace:
I20251211 08:57:40.702306 15318 raft_consensus.cc:2749] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:40.703231 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 1c02cb6ec01041dba118683a57aecde2
I20251211 08:57:40.704751 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c02cb6ec01041dba118683a57aecde2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:40.705324 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c02cb6ec01041dba118683a57aecde2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:40.707333 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c02cb6ec01041dba118683a57aecde2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:40.710536 15083 raft_consensus.cc:493] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:40.711086 15083 raft_consensus.cc:515] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.712939 15083 leader_election.cc:290] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:40.713621 15030 tablet_bootstrap.cc:654] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:40.714221 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dca342fd9de241b5884600199f06fbeb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:40.715370 14385 leader_election.cc:343] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.702153 14905 rpcz_store.cc:276] 1211 08:57:38.531496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.531592 (+ 96us) service_pool.cc:225] Handling call
1211 08:57:40.701560 (+2169968us) ts_tablet_manager.cc:1151] Deleting tablet dca342fd9de241b5884600199f06fbeb
1211 08:57:40.701595 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.701822 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2169482}
I20251211 08:57:40.716349 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dca342fd9de241b5884600199f06fbeb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:40.717339 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet dca342fd9de241b5884600199f06fbeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dca342fd9de241b5884600199f06fbeb already in progress: creating tablet
W20251211 08:57:40.717458 14382 leader_election.cc:343] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.717772 14382 leader_election.cc:304] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:40.718583 15083 raft_consensus.cc:2749] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:40.719400 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 205) took 2187 ms. Trace:
I20251211 08:57:40.719692 14905 rpcz_store.cc:276] 1211 08:57:38.531776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.531864 (+ 88us) service_pool.cc:225] Handling call
1211 08:57:40.717005 (+2185141us) ts_tablet_manager.cc:1151] Deleting tablet 8e05628011634149b17d298c54074d00
1211 08:57:40.717041 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.719337 (+ 2296us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2184641}
I20251211 08:57:40.720719 14905 tablet_replica.cc:333] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:40.721385 14905 raft_consensus.cc:2243] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:40.721832 14905 raft_consensus.cc:2272] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:40.587224 15377 tablet.cc:1722] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:40.723567 14714 raft_consensus.cc:385] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:40.723860 14714 raft_consensus.cc:740] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
W20251211 08:57:40.723933 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 8e05628011634149b17d298c54074d00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e05628011634149b17d298c54074d00 already in progress: creating tablet
I20251211 08:57:40.724472 14714 consensus_queue.cc:260] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.729828 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 211) took 1548 ms. Trace:
I20251211 08:57:40.730361 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e961c5a3b5f640cbaea78b6185048d04" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:40.732610 14385 leader_election.cc:343] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.733163 14011 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_2c5e99b90f0c4c4a8cec03d4471f3364" } new_table_name: "default.Table_50297110fe8f432b9202ae4060127653" modify_external_catalogs: true
I20251211 08:57:40.734277 15193 leader_election.cc:290] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:40.734766 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e961c5a3b5f640cbaea78b6185048d04" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:40.735832 14382 leader_election.cc:343] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.736163 14382 leader_election.cc:304] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:40.736562 14009 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_eb447b641c624aeba08c478e373a532d" } modify_external_catalogs: true
I20251211 08:57:40.730131 14904 rpcz_store.cc:276] 1211 08:57:39.181699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.186333 (+ 4634us) service_pool.cc:225] Handling call
1211 08:57:40.298512 (+1112179us) ts_tablet_manager.cc:1151] Deleting tablet caf537c461c04a81b66f268ee98dc64e
1211 08:57:40.298550 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.499942 (+201392us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.554098 (+ 54156us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.729773 (+175675us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1060526}
I20251211 08:57:40.738510 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_299019253b214d92b585b6e128b24c3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:40.739945 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_299019253b214d92b585b6e128b24c3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:40.741381 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 210) took 1559 ms. Trace:
I20251211 08:57:40.741345 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet caf537c461c04a81b66f268ee98dc64e (table default.Table_ac11e9583ef64c1bbae0b2940213b819 [id=8c183fcd39ae4bbeb0a05bfdd42e755e]) successfully deleted
I20251211 08:57:40.741636 14904 rpcz_store.cc:276] 1211 08:57:39.181493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.181957 (+ 464us) service_pool.cc:225] Handling call
1211 08:57:40.741050 (+1559093us) ts_tablet_manager.cc:1151] Deleting tablet f785a3cc2e8145eebae7e9c2885b0b1e
1211 08:57:40.741094 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.741333 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1470137}
W20251211 08:57:40.742604 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet f785a3cc2e8145eebae7e9c2885b0b1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f785a3cc2e8145eebae7e9c2885b0b1e already in progress: creating tablet
I20251211 08:57:40.748183 14009 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_eb447b641c624aeba08c478e373a532d" } modify_external_catalogs: true
I20251211 08:57:40.750608 15299 ts_tablet_manager.cc:1929] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:40.750950 15299 log.cc:1199] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/db302282bc3d4d8cb56d1cbb2f73c4fe
I20251211 08:57:40.751535 15299 ts_tablet_manager.cc:1950] T db302282bc3d4d8cb56d1cbb2f73c4fe P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:40.757241 14007 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet e8a8437b8b464fb8ab02a9c552f81db3 (table default.Table_b19d002dfb154df6aadac72b8859cce9 [id=0944cc290d954cf4a33fec11acb53315]). Expected version 1 got 0
W20251211 08:57:40.761433 14011 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61775901e28946ddb723f4a01f90feb6 has no consensus state
I20251211 08:57:40.764014 15122 raft_consensus.cc:2749] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:40.765913 14904 tablet_replica.cc:333] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:40.766876 14904 raft_consensus.cc:2243] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:40.767407 14904 raft_consensus.cc:2272] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:40.769358 14714 ts_tablet_manager.cc:1434] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.175s user 0.006s sys 0.000s
I20251211 08:57:40.771180 14905 ts_tablet_manager.cc:1916] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:40.773155 14007 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 40a1f12553934446bfe44f4ed41abf9a (table default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127]). Expected version 1 got 0
I20251211 08:57:40.776681 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 239) took 1305 ms. Trace:
I20251211 08:57:40.776999 15299 rpcz_store.cc:276] 1211 08:57:39.470746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.471378 (+ 632us) service_pool.cc:225] Handling call
1211 08:57:40.562611 (+1091233us) ts_tablet_manager.cc:1151] Deleting tablet db302282bc3d4d8cb56d1cbb2f73c4fe
1211 08:57:40.562655 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.685559 (+122904us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.750412 (+ 64853us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.776628 (+ 26216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090798}
I20251211 08:57:40.778411 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 245) took 1144 ms. Trace:
I20251211 08:57:40.778733 15299 rpcz_store.cc:276] 1211 08:57:39.634068 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.634175 (+ 107us) service_pool.cc:225] Handling call
1211 08:57:40.777934 (+1143759us) ts_tablet_manager.cc:1151] Deleting tablet 9ccfcacb5b294a628926d34cd31b137e
1211 08:57:40.777987 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.778361 (+ 374us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1143295}
I20251211 08:57:40.779897 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 247) took 1109 ms. Trace:
I20251211 08:57:40.779915 14714 tablet_bootstrap.cc:492] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:40.780282 15299 rpcz_store.cc:276] 1211 08:57:39.670656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.670765 (+ 109us) service_pool.cc:225] Handling call
1211 08:57:40.779509 (+1108744us) ts_tablet_manager.cc:1151] Deleting tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0
1211 08:57:40.779580 (+ 71us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.779852 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1108222}
I20251211 08:57:40.781661 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 248) took 1069 ms. Trace:
I20251211 08:57:40.781916 15299 rpcz_store.cc:276] 1211 08:57:39.712050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.712182 (+ 132us) service_pool.cc:225] Handling call
1211 08:57:40.781359 (+1069177us) ts_tablet_manager.cc:1151] Deleting tablet da57b72207414512b9744b283bc869fe
1211 08:57:40.781404 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.781621 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061898}
I20251211 08:57:40.783281 15299 tablet_replica.cc:333] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:40.783938 15299 raft_consensus.cc:2243] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:40.784350 15299 raft_consensus.cc:2272] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:40.786289 15299 ts_tablet_manager.cc:1916] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:40.788763 14904 ts_tablet_manager.cc:1916] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:40.789556 14722 tablet_bootstrap.cc:492] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:40.789908 14722 ts_tablet_manager.cc:1403] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.285s user 0.004s sys 0.007s
I20251211 08:57:40.790076 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet db302282bc3d4d8cb56d1cbb2f73c4fe (table default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]) successfully deleted
W20251211 08:57:40.790812 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 9ccfcacb5b294a628926d34cd31b137e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ccfcacb5b294a628926d34cd31b137e already in progress: creating tablet
W20251211 08:57:40.791342 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 already in progress: creating tablet
W20251211 08:57:40.791862 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet da57b72207414512b9744b283bc869fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da57b72207414512b9744b283bc869fe already in progress: creating tablet
W20251211 08:57:40.792402 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 4e813a4c73334c839bfe6af2119e259c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e813a4c73334c839bfe6af2119e259c already in progress: creating tablet
I20251211 08:57:40.791566 14007 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 14fc87cb76024b5b8227dfe9939e1c89 (table default.Table_fd2daf1c25534c25b34cf363453b0e67 [id=6e2d344612214ee382863680435345b7]). Expected version 1 got 0
I20251211 08:57:40.810271 15030 tablet_bootstrap.cc:492] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:40.810726 15030 ts_tablet_manager.cc:1403] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.167s user 0.004s sys 0.004s
I20251211 08:57:40.817101 14722 raft_consensus.cc:359] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.817672 14722 raft_consensus.cc:385] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:40.817898 14722 raft_consensus.cc:740] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:40.799950 14714 tablet_bootstrap.cc:654] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:40.837095 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 71682.6us
I20251211 08:57:40.837395 15030 raft_consensus.cc:359] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.837952 15030 raft_consensus.cc:385] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:40.838171 15030 raft_consensus.cc:740] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:40.794162 14007 catalog_manager.cc:5654] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:40.841444 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_38d82f258eab426daa74a3a31f588767" } new_table_name: "default.Table_8216a7a3e6804a4f968cb39fc2c20793" modify_external_catalogs: true
I20251211 08:57:40.841564 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2b781e76935e454bb5b396297d73cda4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:40.843051 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b781e76935e454bb5b396297d73cda4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:40.847777 14298 raft_consensus.cc:3060] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:40.863781 15030 consensus_queue.cc:260] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.867024 14722 consensus_queue.cc:260] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.870380 15030 ts_tablet_manager.cc:1434] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.059s user 0.004s sys 0.001s
I20251211 08:57:40.874748 14722 ts_tablet_manager.cc:1434] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.085s user 0.007s sys 0.001s
I20251211 08:57:40.875658 14722 tablet_bootstrap.cc:492] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:40.883047 15318 consensus_queue.cc:1048] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:40.884759 15299 ts_tablet_manager.cc:1929] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:40.885182 15299 log.cc:1199] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/a6570281e1064c79b6e2297723f28658
I20251211 08:57:40.890753 15386 tablet.cc:1722] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:40.891386 15299 ts_tablet_manager.cc:1950] T a6570281e1064c79b6e2297723f28658 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:40.891237 15030 tablet_bootstrap.cc:492] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:40.894093 14714 tablet_bootstrap.cc:492] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:40.894462 14714 ts_tablet_manager.cc:1403] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.115s user 0.006s sys 0.003s
I20251211 08:57:40.897460 14714 raft_consensus.cc:359] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.898380 14714 raft_consensus.cc:385] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:40.907524 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_26f4e336df6947149cb311c15bee56c1" } modify_external_catalogs: true
I20251211 08:57:40.908020 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_26f4e336df6947149cb311c15bee56c1" } modify_external_catalogs: true
I20251211 08:57:40.907239 15318 raft_consensus.cc:493] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:40.909487 15318 raft_consensus.cc:515] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.912057 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b84ea7e1b576445e89c3d3fbd3ed631c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:40.912417 15318 leader_election.cc:290] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
W20251211 08:57:40.913421 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b84ea7e1b576445e89c3d3fbd3ed631c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:40.914286 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_07ea506691994cd3a2dc9606ed81de3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:40.915637 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07ea506691994cd3a2dc9606ed81de3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:40.919107 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "174fefd70bcc41418a77a5716f97cd5f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:40.920158 14544 leader_election.cc:343] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.926143 14722 tablet_bootstrap.cc:654] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:40.926098 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet a6570281e1064c79b6e2297723f28658 (table default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b [id=45f67c67257e428f8ed2f16898ad5fe6]) successfully deleted
W20251211 08:57:40.934875 14544 consensus_peers.cc:597] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:40.945295 15030 tablet_bootstrap.cc:654] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:40.949678 14012 master_service.cc:954] ReplaceTablet: received request to replace tablet eb5a6e533e1f4da7b03a800d029f712d from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:40.964515 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e66c701250064c388e221a300dac3dd0" } new_table_name: "default.Table_53b4b18528d343018f204b9a3b0f9e63" modify_external_catalogs: true
I20251211 08:57:40.978086 14905 ts_tablet_manager.cc:1929] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:40.978471 14905 log.cc:1199] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/9168007ca0584954a734e2c8f2e73f70
I20251211 08:57:40.979169 14905 ts_tablet_manager.cc:1950] T 9168007ca0584954a734e2c8f2e73f70 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:40.979655 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 4e410da32b3446edbf84b2eaa96424c3
I20251211 08:57:40.924387 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "174fefd70bcc41418a77a5716f97cd5f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:40.981328 14447 raft_consensus.cc:2468] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:40.898733 14714 raft_consensus.cc:740] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:40.982331 14542 leader_election.cc:304] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:40.982406 14714 consensus_queue.cc:260] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.983341 14422 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e410da32b3446edbf84b2eaa96424c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:40.983919 15318 raft_consensus.cc:2804] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:40.984225 15318 raft_consensus.cc:493] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:40.984532 15318 raft_consensus.cc:3060] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:40.925863 15299 tablet_replica.cc:333] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:40.949783 15122 raft_consensus.cc:493] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:40.986768 15299 raft_consensus.cc:2243] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:40.945511 15193 raft_consensus.cc:493] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:40.988771 15193 raft_consensus.cc:515] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.992205 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c53035e83710403bac3a2b401c8c97ad" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:40.993070 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c53035e83710403bac3a2b401c8c97ad" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:40.994225 14382 leader_election.cc:343] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:40.995085 14385 leader_election.cc:343] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.995405 14385 leader_election.cc:304] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:40.986526 15122 raft_consensus.cc:515] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:40.997198 15122 leader_election.cc:290] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:40.998349 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "578404732eef4e51b66b6aebffa87cdb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:41.001850 14382 leader_election.cc:343] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:40.990669 15193 leader_election.cc:290] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:41.004063 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e410da32b3446edbf84b2eaa96424c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:41.004801 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e410da32b3446edbf84b2eaa96424c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:41.005071 14722 tablet_bootstrap.cc:492] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:41.005481 14722 ts_tablet_manager.cc:1403] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.130s user 0.007s sys 0.002s
I20251211 08:57:41.006703 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2c8714fcc0964e3c8c96636192b9d6e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:41.007627 14722 raft_consensus.cc:359] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.008213 14722 raft_consensus.cc:385] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251211 08:57:41.008370 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c8714fcc0964e3c8c96636192b9d6e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:41.008456 14722 raft_consensus.cc:740] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:41.009986 15299 raft_consensus.cc:2272] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:41.011240 15083 raft_consensus.cc:2749] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:41.011751 15299 ts_tablet_manager.cc:1916] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:41.012249 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "578404732eef4e51b66b6aebffa87cdb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:41.012987 14714 ts_tablet_manager.cc:1434] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.118s user 0.007s sys 0.000s
W20251211 08:57:41.014019 14385 leader_election.cc:343] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:41.014359 14385 leader_election.cc:304] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:41.015650 15083 raft_consensus.cc:2749] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:41.018847 14722 consensus_queue.cc:260] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:41.028721 13999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20251211 08:57:41.034900 14722 ts_tablet_manager.cc:1434] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20251211 08:57:41.045207 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet eb5a6e533e1f4da7b03a800d029f712d
I20251211 08:57:41.046458 14012 catalog_manager.cc:6724] ReplaceTablet: tablet eb5a6e533e1f4da7b03a800d029f712d deleted and replaced by tablet f6fb2096f80a43b5a5a6230a824fceb7
I20251211 08:57:41.049296 14422 tablet_service.cc:1552] Processing DeleteTablet for tablet eb5a6e533e1f4da7b03a800d029f712d with delete_type TABLET_DATA_DELETED (replaced by tablet f6fb2096f80a43b5a5a6230a824fceb7) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:41.049993 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet eb5a6e533e1f4da7b03a800d029f712d with delete_type TABLET_DATA_DELETED (replaced by tablet f6fb2096f80a43b5a5a6230a824fceb7) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:41.051895 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet eb5a6e533e1f4da7b03a800d029f712d with delete_type TABLET_DATA_DELETED (replaced by tablet f6fb2096f80a43b5a5a6230a824fceb7) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:41.055190 14722 tablet_bootstrap.cc:492] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:41.058712 15318 raft_consensus.cc:515] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.076422 15318 leader_election.cc:290] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:41.077427 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "174fefd70bcc41418a77a5716f97cd5f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:41.077955 14447 raft_consensus.cc:3060] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:41.078168 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 206) took 2420 ms. Trace:
I20251211 08:57:41.078512 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "174fefd70bcc41418a77a5716f97cd5f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:41.078472 14905 rpcz_store.cc:276] 1211 08:57:38.657851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.842859 (+185008us) service_pool.cc:225] Handling call
1211 08:57:40.720604 (+1877745us) ts_tablet_manager.cc:1151] Deleting tablet 9168007ca0584954a734e2c8f2e73f70
1211 08:57:40.720638 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.824523 (+103885us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.977893 (+153370us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:41.078117 (+100224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1877143}
I20251211 08:57:41.080809 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 9168007ca0584954a734e2c8f2e73f70 (table default.Table_b997f016aa90456c8c6b1a4dcf5f4c75 [id=dece67928a044390a750633247cfc2cd]) successfully deleted
I20251211 08:57:40.942597 14904 ts_tablet_manager.cc:1929] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:41.081247 14904 log.cc:1199] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/1976d584cc0a450ab809ead258a251e2
I20251211 08:57:41.081873 14904 ts_tablet_manager.cc:1950] T 1976d584cc0a450ab809ead258a251e2 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:41.082458 15030 tablet_bootstrap.cc:492] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:41.082844 15030 ts_tablet_manager.cc:1403] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.194s user 0.011s sys 0.001s
I20251211 08:57:41.085019 15030 raft_consensus.cc:359] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.085619 15030 raft_consensus.cc:385] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:41.085875 15030 raft_consensus.cc:740] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
W20251211 08:57:41.087044 14544 leader_election.cc:343] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:41.090476 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f2a748aebc4d47738c1758ed749c0f98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:41.086458 15030 consensus_queue.cc:260] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:41.091969 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2a748aebc4d47738c1758ed749c0f98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:41.093511 15030 ts_tablet_manager.cc:1434] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251211 08:57:41.095185 14447 raft_consensus.cc:2468] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:41.096349 14542 leader_election.cc:304] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:41.098158 15152 raft_consensus.cc:2804] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:41.098591 15152 raft_consensus.cc:697] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:41.099395 15152 consensus_queue.cc:237] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.101089 14722 tablet_bootstrap.cc:654] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:41.102852 14714 tablet_bootstrap.cc:492] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:41.103854 15030 tablet_bootstrap.cc:492] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:41.104154 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 825d8cf806d94702bdd11e0d63ee0c59 (DEFAULT_TABLE table=default.Table_5f7ac9a06e714b2bb4790efd948dc9ee [id=14a7bb4413b2479f9e30f4a430e8ab90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.105125 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825d8cf806d94702bdd11e0d63ee0c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.119681 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet e8a8437b8b464fb8ab02a9c552f81db3 (table default.Table_b19d002dfb154df6aadac72b8859cce9 [id=0944cc290d954cf4a33fec11acb53315]). Expected version 1 got 0
I20251211 08:57:41.120720 15030 tablet_bootstrap.cc:654] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:41.121562 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 78c6a703e1e44f41931fe465def01bf6 (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]). Expected version 2 got 0
I20251211 08:57:41.127211 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 40a1f12553934446bfe44f4ed41abf9a (table default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127]). Expected version 1 got 0
I20251211 08:57:41.128878 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 41d6ef975a47489385791065bb8f9f46 (table default.Table_c9d62497a342488290749d4f7c02e748 [id=52ae4637827d424eb4d37362bf21cea2]). Expected version 2 got 0
I20251211 08:57:41.129384 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 14fc87cb76024b5b8227dfe9939e1c89 (table default.Table_fd2daf1c25534c25b34cf363453b0e67 [id=6e2d344612214ee382863680435345b7]). Expected version 1 got 0
I20251211 08:57:41.130304 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 578404732eef4e51b66b6aebffa87cdb (table default.Table_e66c701250064c388e221a300dac3dd0 [id=26c0a9d9b1f84372b18a54f045eb768c]). Expected version 1 got 0
I20251211 08:57:41.130882 14422 tablet_service.cc:1505] Processing CreateTablet for tablet 825d8cf806d94702bdd11e0d63ee0c59 (DEFAULT_TABLE table=default.Table_5f7ac9a06e714b2bb4790efd948dc9ee [id=14a7bb4413b2479f9e30f4a430e8ab90]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:41.131994 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 578404732eef4e51b66b6aebffa87cdb consensus state has no leader
I20251211 08:57:41.132154 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825d8cf806d94702bdd11e0d63ee0c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.133493 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 40a1f12553934446bfe44f4ed41abf9a (table default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127]). Expected version 1 got 0
I20251211 08:57:41.134246 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 14fc87cb76024b5b8227dfe9939e1c89 (table default.Table_fd2daf1c25534c25b34cf363453b0e67 [id=6e2d344612214ee382863680435345b7]). Expected version 1 got 0
I20251211 08:57:41.137410 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 7af7c9ab27a64a1681f9874a922e986f (DEFAULT_TABLE table=default.Table_d9c4c3852105418a914f5055b9b70072 [id=60f08078e7f849c09040202347df4f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.138527 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7af7c9ab27a64a1681f9874a922e986f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.108845 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 207) took 2253 ms. Trace:
I20251211 08:57:41.142228 14905 rpcz_store.cc:276] 1211 08:57:38.855935 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:38.871603 (+ 15668us) service_pool.cc:225] Handling call
1211 08:57:41.108511 (+2236908us) ts_tablet_manager.cc:1151] Deleting tablet e9905227c0a844d199362a442e7d630c
1211 08:57:41.108552 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.108787 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2236361}
I20251211 08:57:41.143383 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 209) took 2082 ms. Trace:
I20251211 08:57:41.144564 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet 81cfb3e0baef45ddab4a77a4e0790085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:41.144933 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet e9905227c0a844d199362a442e7d630c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9905227c0a844d199362a442e7d630c already in progress: creating tablet
I20251211 08:57:41.157704 15299 ts_tablet_manager.cc:1929] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:41.158062 15299 log.cc:1199] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/8ad600554c0946e0bcb0da6a42be2ffd
I20251211 08:57:41.158165 14714 tablet_bootstrap.cc:654] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:41.161530 14010 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 080d1700df4a4df09acda65c6d021d96
I20251211 08:57:41.163928 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 080d1700df4a4df09acda65c6d021d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:41.165277 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 080d1700df4a4df09acda65c6d021d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:41.143673 14905 rpcz_store.cc:276] 1211 08:57:39.060516 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.060629 (+ 113us) service_pool.cc:225] Handling call
1211 08:57:41.143076 (+2082447us) ts_tablet_manager.cc:1151] Deleting tablet f785a3cc2e8145eebae7e9c2885b0b1e
1211 08:57:41.143117 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.143336 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2081976}
I20251211 08:57:41.167843 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet 080d1700df4a4df09acda65c6d021d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:41.168506 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet f785a3cc2e8145eebae7e9c2885b0b1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f785a3cc2e8145eebae7e9c2885b0b1e already in progress: creating tablet
I20251211 08:57:41.169837 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 210) took 2094 ms. Trace:
I20251211 08:57:41.170110 14905 rpcz_store.cc:276] 1211 08:57:39.075763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.078396 (+ 2633us) service_pool.cc:225] Handling call
1211 08:57:41.169533 (+2091137us) ts_tablet_manager.cc:1151] Deleting tablet f3403116517f45c59f87fa5bda0b4071
1211 08:57:41.169572 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.169789 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2090484}
W20251211 08:57:41.171284 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet f3403116517f45c59f87fa5bda0b4071 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3403116517f45c59f87fa5bda0b4071 already in progress: creating tablet
I20251211 08:57:41.179550 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 222) took 1536 ms. Trace:
I20251211 08:57:41.179854 14905 rpcz_store.cc:276] 1211 08:57:39.643059 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.643153 (+ 94us) service_pool.cc:225] Handling call
1211 08:57:41.179144 (+1535991us) ts_tablet_manager.cc:1151] Deleting tablet 9ccfcacb5b294a628926d34cd31b137e
1211 08:57:41.179202 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.179502 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1535431}
W20251211 08:57:41.180898 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 9ccfcacb5b294a628926d34cd31b137e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ccfcacb5b294a628926d34cd31b137e already in progress: creating tablet
I20251211 08:57:41.082548 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 226) took 1405 ms. Trace:
I20251211 08:57:41.114418 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 825d8cf806d94702bdd11e0d63ee0c59 (DEFAULT_TABLE table=default.Table_5f7ac9a06e714b2bb4790efd948dc9ee [id=14a7bb4413b2479f9e30f4a430e8ab90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.186914 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825d8cf806d94702bdd11e0d63ee0c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.191349 14905 tablet_replica.cc:333] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:41.193185 14905 raft_consensus.cc:2243] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:41.196223 14905 raft_consensus.cc:2272] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:41.199044 15368 raft_consensus.cc:493] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:41.199501 15368 raft_consensus.cc:515] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.158715 15299 ts_tablet_manager.cc:1950] T 8ad600554c0946e0bcb0da6a42be2ffd P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:41.201349 15368 leader_election.cc:290] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:41.202962 14230 leader_election.cc:343] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 81cfb3e0baef45ddab4a77a4e0790085
W20251211 08:57:41.204263 14231 leader_election.cc:343] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 81cfb3e0baef45ddab4a77a4e0790085
I20251211 08:57:41.204614 14231 leader_election.cc:304] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:41.205796 15363 raft_consensus.cc:2749] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:41.199525 14904 rpcz_store.cc:276] 1211 08:57:39.677210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.677325 (+ 115us) service_pool.cc:225] Handling call
1211 08:57:40.765695 (+1088370us) ts_tablet_manager.cc:1151] Deleting tablet 1976d584cc0a450ab809ead258a251e2
1211 08:57:40.765766 (+ 71us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:40.842088 (+ 76322us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:40.942402 (+100314us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:41.082500 (+140098us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1087874}
I20251211 08:57:41.211501 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 1976d584cc0a450ab809ead258a251e2 (table default.Table_81eee4b23e9045848867df8a0f511ae5 [id=72d8518a3dbf4dfdb28b1f422e44d414]) successfully deleted
I20251211 08:57:41.211654 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 7af7c9ab27a64a1681f9874a922e986f (DEFAULT_TABLE table=default.Table_d9c4c3852105418a914f5055b9b70072 [id=60f08078e7f849c09040202347df4f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.212652 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7af7c9ab27a64a1681f9874a922e986f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.212993 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 7af7c9ab27a64a1681f9874a922e986f (DEFAULT_TABLE table=default.Table_d9c4c3852105418a914f5055b9b70072 [id=60f08078e7f849c09040202347df4f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.213902 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7af7c9ab27a64a1681f9874a922e986f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.216212 14586 tablet_service.cc:1505] Processing CreateTablet for tablet b5884dc6ef814552893cdf9dccfcce19 (DEFAULT_TABLE table=default.Table_939cb0831eff4bdebbb7fc6d0052a06a [id=47f1741e4d8342cb9afb0a268e4af52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.217036 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5884dc6ef814552893cdf9dccfcce19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.219734 14426 tablet_service.cc:1505] Processing CreateTablet for tablet b5884dc6ef814552893cdf9dccfcce19 (DEFAULT_TABLE table=default.Table_939cb0831eff4bdebbb7fc6d0052a06a [id=47f1741e4d8342cb9afb0a268e4af52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.220665 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5884dc6ef814552893cdf9dccfcce19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.220692 15193 raft_consensus.cc:493] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:41.221138 15193 raft_consensus.cc:515] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.222743 15193 leader_election.cc:290] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:41.224011 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b5a4b6b1b28406c82e2acd9356b60d4" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:41.225108 14382 leader_election.cc:343] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:41.230661 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 227) took 1551 ms. Trace:
I20251211 08:57:41.230985 14004 master_service.cc:954] ReplaceTablet: received request to replace tablet ac6be204692a44fab2db551e79237299 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:41.231004 14904 rpcz_store.cc:276] 1211 08:57:39.679421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.679538 (+ 117us) service_pool.cc:225] Handling call
1211 08:57:41.230185 (+1550647us) ts_tablet_manager.cc:1151] Deleting tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0
1211 08:57:41.230351 (+ 166us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.230601 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1550188}
I20251211 08:57:41.232177 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 225) took 1556 ms. Trace:
I20251211 08:57:41.232467 14904 rpcz_store.cc:276] 1211 08:57:39.675506 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.682745 (+ 7239us) service_pool.cc:225] Handling call
1211 08:57:41.231879 (+1549134us) ts_tablet_manager.cc:1151] Deleting tablet 9ccfcacb5b294a628926d34cd31b137e
1211 08:57:41.231930 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.232135 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1546672}
I20251211 08:57:41.233449 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 228) took 1502 ms. Trace:
I20251211 08:57:41.233717 14904 rpcz_store.cc:276] 1211 08:57:39.730610 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.734653 (+ 4043us) service_pool.cc:225] Handling call
1211 08:57:41.233170 (+1498517us) ts_tablet_manager.cc:1151] Deleting tablet da57b72207414512b9744b283bc869fe
1211 08:57:41.233203 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.233409 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1451707}
I20251211 08:57:41.241510 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d40cb8a6b4974e2e9ff982e893a2c552" } modify_external_catalogs: true
I20251211 08:57:41.242085 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d40cb8a6b4974e2e9ff982e893a2c552" } modify_external_catalogs: true
I20251211 08:57:41.244026 15299 tablet_replica.cc:333] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:41.245000 15299 raft_consensus.cc:2243] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:41.245579 15299 raft_consensus.cc:2272] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:41.246863 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 8ad600554c0946e0bcb0da6a42be2ffd (table default.Table_ccb3af786cad4564b032d78520da1cc3 [id=62e4ec0c216b41eebee5490eaba6ef5f]) successfully deleted
I20251211 08:57:41.247519 14007 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9] alter complete (version 2)
I20251211 08:57:41.247656 14006 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_4ab19ee3f1f3473fa3d320e7841a201c [id=4208ab42111742f38028caeb7913a1c7] alter complete (version 1)
I20251211 08:57:41.250813 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_e1af8fbbea91449781ac2466fe85e5c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:41.252316 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1af8fbbea91449781ac2466fe85e5c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:41.263906 14267 tablet_service.cc:1505] Processing CreateTablet for tablet b5884dc6ef814552893cdf9dccfcce19 (DEFAULT_TABLE table=default.Table_939cb0831eff4bdebbb7fc6d0052a06a [id=47f1741e4d8342cb9afb0a268e4af52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.265158 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5884dc6ef814552893cdf9dccfcce19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.267995 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b5a4b6b1b28406c82e2acd9356b60d4" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:41.269202 15368 raft_consensus.cc:493] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:41.269773 15368 raft_consensus.cc:515] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.271782 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 4dcd09f7c6364d74bb33503b5bbfea96 (DEFAULT_TABLE table=default.Table_0442aa229734421dab4cf2f058c9c5e0 [id=482af2b30df84c8891e037e1aea436af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.272051 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_010b1e9526c341ef81e1805cb6929e30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:41.272862 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dcd09f7c6364d74bb33503b5bbfea96. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:41.273214 14385 leader_election.cc:343] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:41.273499 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_010b1e9526c341ef81e1805cb6929e30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:41.273571 14385 leader_election.cc:304] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:41.274628 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f826d079774152a3b66bfb81f990a8" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:41.276348 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 4dcd09f7c6364d74bb33503b5bbfea96 (DEFAULT_TABLE table=default.Table_0442aa229734421dab4cf2f058c9c5e0 [id=482af2b30df84c8891e037e1aea436af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.277384 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dcd09f7c6364d74bb33503b5bbfea96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.278910 15122 raft_consensus.cc:2749] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:41.279836 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 already in progress: creating tablet
I20251211 08:57:41.280105 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 4dcd09f7c6364d74bb33503b5bbfea96 (DEFAULT_TABLE table=default.Table_0442aa229734421dab4cf2f058c9c5e0 [id=482af2b30df84c8891e037e1aea436af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.281034 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dcd09f7c6364d74bb33503b5bbfea96. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:41.280562 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 9ccfcacb5b294a628926d34cd31b137e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ccfcacb5b294a628926d34cd31b137e already in progress: creating tablet
W20251211 08:57:41.282742 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet da57b72207414512b9744b283bc869fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da57b72207414512b9744b283bc869fe already in progress: creating tablet
I20251211 08:57:41.283872 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_84155c7aff0d471687b30b86e434adf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:41.285228 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84155c7aff0d471687b30b86e434adf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:41.315517 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 9168007ca0584954a734e2c8f2e73f70 with delete_type TABLET_DATA_DELETED (replaced by tablet f56fd93aeac6435ea84388ce05580113) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:41.316140 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 81cfb3e0baef45ddab4a77a4e0790085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:41.316100 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f826d079774152a3b66bfb81f990a8" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:41.292102 15368 leader_election.cc:290] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:41.316900 15368 raft_consensus.cc:493] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:41.317456 14231 leader_election.cc:304] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:41.317305 15368 raft_consensus.cc:515] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.318516 15363 raft_consensus.cc:2749] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:41.319519 15368 leader_election.cc:290] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:41.320837 14231 leader_election.cc:343] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: b8d753358f20476ba23d7d7b72e26e74
W20251211 08:57:41.321434 14230 leader_election.cc:343] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: b8d753358f20476ba23d7d7b72e26e74
I20251211 08:57:41.321591 14722 tablet_bootstrap.cc:492] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:41.321782 14230 leader_election.cc:304] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:41.321995 14722 ts_tablet_manager.cc:1403] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.267s user 0.008s sys 0.001s
I20251211 08:57:41.324137 14722 raft_consensus.cc:359] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.324709 14722 raft_consensus.cc:385] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:41.324592 14009 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet d4f826d079774152a3b66bfb81f990a8 (table default.Table_4ab19ee3f1f3473fa3d320e7841a201c [id=4208ab42111742f38028caeb7913a1c7]). Expected version 1 got 0
I20251211 08:57:41.324973 14722 raft_consensus.cc:740] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:41.327400 15368 raft_consensus.cc:2749] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:41.333585 15030 tablet_bootstrap.cc:492] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:41.334005 15030 ts_tablet_manager.cc:1403] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.231s user 0.011s sys 0.028s
I20251211 08:57:41.336045 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet db302282bc3d4d8cb56d1cbb2f73c4fe with delete_type TABLET_DATA_DELETED (replaced by tablet f785a3cc2e8145eebae7e9c2885b0b1e) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:41.336228 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet ac6be204692a44fab2db551e79237299
I20251211 08:57:41.336644 15030 raft_consensus.cc:359] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.337426 14004 catalog_manager.cc:6724] ReplaceTablet: tablet ac6be204692a44fab2db551e79237299 deleted and replaced by tablet 2c0f8d7dd90f4beaae674fe827d37e0b
I20251211 08:57:41.337250 15030 raft_consensus.cc:385] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:41.341046 15030 raft_consensus.cc:740] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:41.341766 15030 consensus_queue.cc:260] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.342885 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet ac6be204692a44fab2db551e79237299 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c0f8d7dd90f4beaae674fe827d37e0b) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:41.347184 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9d16e03fa298473cb4acdb00f6143d5e" } new_table_name: "default.Table_bdaa13035e5f4ecb80d93694ed51149e" modify_external_catalogs: true
I20251211 08:57:41.347668 14714 tablet_bootstrap.cc:492] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:41.348052 14714 ts_tablet_manager.cc:1403] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.334s user 0.006s sys 0.005s
I20251211 08:57:41.348006 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet ac6be204692a44fab2db551e79237299 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c0f8d7dd90f4beaae674fe827d37e0b) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:41.353713 15030 ts_tablet_manager.cc:1434] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.019s user 0.002s sys 0.005s
I20251211 08:57:41.197758 14905 ts_tablet_manager.cc:1916] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:41.354848 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_2b781e76935e454bb5b396297d73cda4" } new_table_name: "default.Table_0e736d3e3e5f40dbbf25a5e615e5b842" modify_external_catalogs: true
I20251211 08:57:41.355701 14722 consensus_queue.cc:260] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.234690 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 238) took 1081 ms. Trace:
I20251211 08:57:41.367165 14904 rpcz_store.cc:276] 1211 08:57:40.153374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.153476 (+ 102us) service_pool.cc:225] Handling call
1211 08:57:41.234393 (+1080917us) ts_tablet_manager.cc:1151] Deleting tablet 4e813a4c73334c839bfe6af2119e259c
1211 08:57:41.234453 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.234650 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1080389}
I20251211 08:57:41.356199 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet ac6be204692a44fab2db551e79237299 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c0f8d7dd90f4beaae674fe827d37e0b) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:41.362411 15299 ts_tablet_manager.cc:1916] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:41.371450 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 4e813a4c73334c839bfe6af2119e259c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e813a4c73334c839bfe6af2119e259c already in progress: creating tablet
I20251211 08:57:41.373636 14298 raft_consensus.cc:3060] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:41.367174 15416 tablet.cc:1722] T 383716a1f1f847ccbea542eca04790c9 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:41.370290 14722 ts_tablet_manager.cc:1434] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.048s user 0.006s sys 0.001s
I20251211 08:57:41.385249 15030 tablet_bootstrap.cc:492] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:41.391206 15030 tablet_bootstrap.cc:654] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:41.406999 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ad600554c0946e0bcb0da6a42be2ffd with delete_type TABLET_DATA_DELETED (replaced by tablet 6874fb018bc844b3aeeab7db8fc53e9a) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:41.407805 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet db302282bc3d4d8cb56d1cbb2f73c4fe with delete_type TABLET_DATA_DELETED (replaced by tablet f785a3cc2e8145eebae7e9c2885b0b1e) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:41.412967 15030 tablet_bootstrap.cc:492] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:41.413384 15030 ts_tablet_manager.cc:1403] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.000s
I20251211 08:57:41.415549 15030 raft_consensus.cc:359] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.416163 15030 raft_consensus.cc:385] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:41.416414 15030 raft_consensus.cc:740] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:41.417178 15030 consensus_queue.cc:260] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.419498 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 680c8f54e75a4291bb4d769d94e21552 (DEFAULT_TABLE table=default.Table_f1a804079b754f0e84b85a175c773a3f [id=387069b529984870961fe2e545fbb221]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.420709 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680c8f54e75a4291bb4d769d94e21552. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.422870 14904 tablet_replica.cc:333] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: stopping tablet replica
W20251211 08:57:41.423045 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 49015c5d4f2b4d07b547a9df57c992eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49015c5d4f2b4d07b547a9df57c992eb already in progress: creating tablet
I20251211 08:57:41.423434 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 7264e13dd44b499683e32b3fc62ed8ed (DEFAULT_TABLE table=default.Table_182b8f62776541cea88982532ffe9fcb [id=44ce3a15badc4db39f1c6c67e2d906d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.423590 14904 raft_consensus.cc:2243] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:41.424083 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 7264e13dd44b499683e32b3fc62ed8ed (DEFAULT_TABLE table=default.Table_182b8f62776541cea88982532ffe9fcb [id=44ce3a15badc4db39f1c6c67e2d906d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.424433 14904 pending_rounds.cc:70] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
I20251211 08:57:41.424558 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7264e13dd44b499683e32b3fc62ed8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.424678 14904 pending_rounds.cc:77] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256414646398976 op_type: NO_OP noop_request { }
I20251211 08:57:41.424973 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7264e13dd44b499683e32b3fc62ed8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.427191 14722 tablet_bootstrap.cc:492] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:41.430332 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 7264e13dd44b499683e32b3fc62ed8ed (DEFAULT_TABLE table=default.Table_182b8f62776541cea88982532ffe9fcb [id=44ce3a15badc4db39f1c6c67e2d906d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.431317 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7264e13dd44b499683e32b3fc62ed8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.434274 15417 tablet.cc:1722] T 383716a1f1f847ccbea542eca04790c9 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:41.443090 15030 ts_tablet_manager.cc:1434] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20251211 08:57:41.424984 14904 raft_consensus.cc:2889] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:41.443831 14904 raft_consensus.cc:2272] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:41.444382 15030 tablet_bootstrap.cc:492] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:41.445888 14904 ts_tablet_manager.cc:1916] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:41.391729 14714 raft_consensus.cc:359] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.454393 14714 raft_consensus.cc:385] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251211 08:57:41.456367 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb consensus state has no leader
I20251211 08:57:41.456295 15386 tablet.cc:1722] T d4f826d079774152a3b66bfb81f990a8 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:41.467324 14422 tablet_service.cc:1505] Processing CreateTablet for tablet 680c8f54e75a4291bb4d769d94e21552 (DEFAULT_TABLE table=default.Table_f1a804079b754f0e84b85a175c773a3f [id=387069b529984870961fe2e545fbb221]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.468389 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680c8f54e75a4291bb4d769d94e21552. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.469115 15030 tablet_bootstrap.cc:654] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:41.476107 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 680c8f54e75a4291bb4d769d94e21552 (DEFAULT_TABLE table=default.Table_f1a804079b754f0e84b85a175c773a3f [id=387069b529984870961fe2e545fbb221]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.477283 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680c8f54e75a4291bb4d769d94e21552. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.479676 14714 raft_consensus.cc:740] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:41.480222 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet a6570281e1064c79b6e2297723f28658 with delete_type TABLET_DATA_DELETED (replaced by tablet e9905227c0a844d199362a442e7d630c) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:41.480369 14714 consensus_queue.cc:260] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.482967 15368 raft_consensus.cc:493] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:41.483457 15368 raft_consensus.cc:515] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:41.488229 14231 leader_election.cc:343] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 42720f5f5bf445f2b72e513ad4a6d95f
I20251211 08:57:41.489907 15368 leader_election.cc:290] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:41.481587 15386 tablet.cc:1722] T 383716a1f1f847ccbea542eca04790c9 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:41.492465 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42720f5f5bf445f2b72e513ad4a6d95f" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:41.493757 14230 leader_election.cc:304] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:41.494607 15368 raft_consensus.cc:2749] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:41.499588 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 42720f5f5bf445f2b72e513ad4a6d95f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:41.499817 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet dca342fd9de241b5884600199f06fbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:41.501920 14426 tablet_service.cc:1505] Processing CreateTablet for tablet a74d250ee85a4c768c76a8400003ccdb (DEFAULT_TABLE table=default.Table_417cd1cc5e354dddaa9d5c0c3f8ddd8c [id=86c1d7cba4584192b83c91c38bcfa23e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.502835 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a74d250ee85a4c768c76a8400003ccdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.502585 14275 tablet_service.cc:1505] Processing CreateTablet for tablet f18036362c93413fa7690906a0298784 (DEFAULT_TABLE table=default.Table_626342780a404385ad91b9e642a77ed8 [id=275815c2da944e178e54afe549de34e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.503593 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18036362c93413fa7690906a0298784. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.503973 14266 tablet_service.cc:1505] Processing CreateTablet for tablet a74d250ee85a4c768c76a8400003ccdb (DEFAULT_TABLE table=default.Table_417cd1cc5e354dddaa9d5c0c3f8ddd8c [id=86c1d7cba4584192b83c91c38bcfa23e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.504973 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a74d250ee85a4c768c76a8400003ccdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.507001 14722 tablet_bootstrap.cc:654] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:41.508503 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_873e2f0afae049eb85e3652c71880de5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:41.509363 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet e961c5a3b5f640cbaea78b6185048d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:41.509908 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_873e2f0afae049eb85e3652c71880de5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:41.518924 14714 ts_tablet_manager.cc:1434] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.149s user 0.007s sys 0.000s
W20251211 08:57:41.520561 14006 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 32) took 894 ms (client timeout 1000 ms). Trace:
I20251211 08:57:41.532756 14587 tablet_service.cc:1505] Processing CreateTablet for tablet a74d250ee85a4c768c76a8400003ccdb (DEFAULT_TABLE table=default.Table_417cd1cc5e354dddaa9d5c0c3f8ddd8c [id=86c1d7cba4584192b83c91c38bcfa23e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.533715 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a74d250ee85a4c768c76a8400003ccdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.544072 15030 tablet_bootstrap.cc:492] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:41.544476 15030 ts_tablet_manager.cc:1403] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.100s user 0.007s sys 0.003s
I20251211 08:57:41.545588 14010 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 07aef1eea7c04d9cbf35c82e45df7a16
I20251211 08:57:41.546533 15030 raft_consensus.cc:359] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.547188 15030 raft_consensus.cc:385] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:41.547425 15030 raft_consensus.cc:740] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:41.547931 15030 consensus_queue.cc:260] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.548698 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:41.550383 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:41.552472 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:41.520884 14006 rpcz_store.cc:269] 1211 08:57:40.626155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.680924 (+ 54769us) service_pool.cc:225] Handling call
1211 08:57:40.714863 (+ 33939us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:41.520487 (+805624us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:41.170177 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:41.170697 (+ 520us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:41.170730 (+ 33us) write_op.cc:435] Acquired schema lock
1211 08:57:41.170748 (+ 18us) tablet.cc:660] Decoding operations
1211 08:57:41.171066 (+ 318us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:41.171095 (+ 29us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:41.171115 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:41.171231 (+ 116us) tablet.cc:699] Row locks acquired
1211 08:57:41.171243 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:57:41.171327 (+ 84us) write_op.cc:270] Start()
1211 08:57:41.171390 (+ 63us) write_op.cc:276] Timestamp: P: 1765443461171386 usec, L: 0
1211 08:57:41.171403 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:41.171775 (+ 372us) log.cc:844] Serialized 356 byte log entry
1211 08:57:41.243270 (+ 71495us) op_driver.cc:464] REPLICATION: finished
1211 08:57:41.245428 (+ 2158us) write_op.cc:301] APPLY: starting
1211 08:57:41.245485 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:41.245554 (+ 69us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:41.245637 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:41.245759 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:41.245866 (+ 107us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:41.245904 (+ 38us) write_op.cc:312] APPLY: finished
1211 08:57:41.246028 (+ 124us) log.cc:844] Serialized 23 byte log entry
1211 08:57:41.246149 (+ 121us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:41.246200 (+ 51us) write_op.cc:454] Released schema lock
1211 08:57:41.246225 (+ 25us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:41.257333 (+ 11108us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:41.257356 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":347,"reported_tablets":18,"spinlock_wait_cycles":63161088,"child_traces":[["op",{"apply.queue_time_us":2038,"apply.run_cpu_time_us":955,"apply.run_wall_time_us":955,"mutex_wait_us":17489,"num_ops":1,"prepare.queue_time_us":28364,"prepare.run_cpu_time_us":1899,"prepare.run_wall_time_us":87183,"raft.queue_time_us":1598,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":13,"replication_time_us":71790}]]}
I20251211 08:57:41.559651 15030 ts_tablet_manager.cc:1434] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20251211 08:57:41.562980 15030 tablet_bootstrap.cc:492] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:41.562805 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_03bee763242749cfa5fb0fe7d7e7fb7f" } modify_external_catalogs: true
I20251211 08:57:41.563382 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_03bee763242749cfa5fb0fe7d7e7fb7f" } modify_external_catalogs: true
I20251211 08:57:41.599395 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 947e6203f910442ca01e2c21587cfe90 (DEFAULT_TABLE table=default.Table_09b494206172425ba445408538dfebcb [id=85e949cd1d0e41888f9839d01eb742dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.599463 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 947e6203f910442ca01e2c21587cfe90 (DEFAULT_TABLE table=default.Table_09b494206172425ba445408538dfebcb [id=85e949cd1d0e41888f9839d01eb742dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.600334 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 947e6203f910442ca01e2c21587cfe90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.600345 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 947e6203f910442ca01e2c21587cfe90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.607510 14420 tablet_service.cc:1505] Processing CreateTablet for tablet f18036362c93413fa7690906a0298784 (DEFAULT_TABLE table=default.Table_626342780a404385ad91b9e642a77ed8 [id=275815c2da944e178e54afe549de34e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.608551 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18036362c93413fa7690906a0298784. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.614167 15299 ts_tablet_manager.cc:1929] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:41.597872 14584 tablet_service.cc:1505] Processing CreateTablet for tablet f18036362c93413fa7690906a0298784 (DEFAULT_TABLE table=default.Table_626342780a404385ad91b9e642a77ed8 [id=275815c2da944e178e54afe549de34e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.615684 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18036362c93413fa7690906a0298784. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.634943 15299 log.cc:1199] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/174fefd70bcc41418a77a5716f97cd5f
I20251211 08:57:41.635671 15299 ts_tablet_manager.cc:1950] T 174fefd70bcc41418a77a5716f97cd5f P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:41.630074 15030 tablet_bootstrap.cc:654] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:41.653015 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 947e6203f910442ca01e2c21587cfe90 (DEFAULT_TABLE table=default.Table_09b494206172425ba445408538dfebcb [id=85e949cd1d0e41888f9839d01eb742dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.653959 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 947e6203f910442ca01e2c21587cfe90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.653757 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 27cb4228204c4176b1a348960efd5b26 (DEFAULT_TABLE table=default.Table_2f325f51dd9f433f98cfef7169cdbe0c [id=2ff95cefc4a545398f979e2bd0321f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.654634 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27cb4228204c4176b1a348960efd5b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.666386 14722 tablet_bootstrap.cc:492] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:41.666889 14722 ts_tablet_manager.cc:1403] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.241s user 0.016s sys 0.003s
I20251211 08:57:41.681474 14714 tablet_bootstrap.cc:492] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:41.691143 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 266) took 1190 ms. Trace:
I20251211 08:57:41.691568 15299 rpcz_store.cc:276] 1211 08:57:40.500583 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.502511 (+ 1928us) service_pool.cc:225] Handling call
1211 08:57:41.243884 (+741373us) ts_tablet_manager.cc:1151] Deleting tablet 174fefd70bcc41418a77a5716f97cd5f
1211 08:57:41.243922 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.447259 (+203337us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:41.613972 (+166713us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:41.691075 (+ 77103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50678,"tablet-delete.queue_time_us":740878}
I20251211 08:57:41.693024 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 265) took 1192 ms. Trace:
I20251211 08:57:41.692986 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 174fefd70bcc41418a77a5716f97cd5f (table default.Table_fcf104ee591d470d806156189c2cac52 [id=b734e827456d451b9fe6f1dbf47fea1d]) successfully deleted
I20251211 08:57:41.693280 15299 rpcz_store.cc:276] 1211 08:57:40.500291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.503406 (+ 3115us) service_pool.cc:225] Handling call
1211 08:57:41.692807 (+1189401us) ts_tablet_manager.cc:1151] Deleting tablet 9168007ca0584954a734e2c8f2e73f70
1211 08:57:41.692842 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.692984 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1188856}
I20251211 08:57:41.694381 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 269) took 1160 ms. Trace:
W20251211 08:57:41.694375 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 9168007ca0584954a734e2c8f2e73f70 because the tablet was not found. No further retry: Not found: Tablet not found: 9168007ca0584954a734e2c8f2e73f70
I20251211 08:57:41.698840 15360 raft_consensus.cc:493] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:41.699457 15360 raft_consensus.cc:515] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:41.704385 14542 leader_election.cc:343] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: a6570281e1064c79b6e2297723f28658
I20251211 08:57:41.705960 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6570281e1064c79b6e2297723f28658" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:41.706691 15360 leader_election.cc:290] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:41.707223 14544 leader_election.cc:343] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:41.708757 14544 leader_election.cc:304] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:41.710106 15360 raft_consensus.cc:2749] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:41.718941 14012 master_service.cc:954] ReplaceTablet: received request to replace tablet 78c6a703e1e44f41931fe465def01bf6 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:41.724012 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 27cb4228204c4176b1a348960efd5b26 (DEFAULT_TABLE table=default.Table_2f325f51dd9f433f98cfef7169cdbe0c [id=2ff95cefc4a545398f979e2bd0321f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.724891 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27cb4228204c4176b1a348960efd5b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.727744 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 143f20f5c37948068ff2fb0d92f0f34b (DEFAULT_TABLE table=default.Table_3ec948949bdb4957b702fc9f5b18207a [id=9eb6a42c3a1e48f69acc0948e22afb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.727850 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 143f20f5c37948068ff2fb0d92f0f34b (DEFAULT_TABLE table=default.Table_3ec948949bdb4957b702fc9f5b18207a [id=9eb6a42c3a1e48f69acc0948e22afb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.728744 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143f20f5c37948068ff2fb0d92f0f34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.728992 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 27cb4228204c4176b1a348960efd5b26 (DEFAULT_TABLE table=default.Table_2f325f51dd9f433f98cfef7169cdbe0c [id=2ff95cefc4a545398f979e2bd0321f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.729795 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 1065085c701f48c9ad7206c410ce5985 (DEFAULT_TABLE table=default.Table_93954115dc5d4d2283faa424e83608b0 [id=cb30302a7db04098ae9a5a92ce055d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.731036 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1065085c701f48c9ad7206c410ce5985. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.731667 15360 raft_consensus.cc:493] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:41.732053 15360 raft_consensus.cc:515] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.733695 15360 leader_election.cc:290] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:41.734367 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dca342fd9de241b5884600199f06fbeb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:41.735019 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dca342fd9de241b5884600199f06fbeb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:41.735524 14449 raft_consensus.cc:2468] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:41.735812 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 1065085c701f48c9ad7206c410ce5985 (DEFAULT_TABLE table=default.Table_93954115dc5d4d2283faa424e83608b0 [id=cb30302a7db04098ae9a5a92ce055d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.728920 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143f20f5c37948068ff2fb0d92f0f34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.736546 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 1065085c701f48c9ad7206c410ce5985 (DEFAULT_TABLE table=default.Table_93954115dc5d4d2283faa424e83608b0 [id=cb30302a7db04098ae9a5a92ce055d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.736876 14542 leader_election.cc:304] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
W20251211 08:57:41.737715 14544 leader_election.cc:343] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:41.737921 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27cb4228204c4176b1a348960efd5b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.739120 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1065085c701f48c9ad7206c410ce5985. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.741863 15360 raft_consensus.cc:2804] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:41.742283 15360 raft_consensus.cc:493] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:41.742539 15360 raft_consensus.cc:3060] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:41.743726 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1065085c701f48c9ad7206c410ce5985. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.749050 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 4c95565ce3954cc598d7cbd45321d7c9 (DEFAULT_TABLE table=default.Table_111bf58f55b84e8088d40992e55dd3f2 [id=34dadde82105442eafc526ce3430760e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.749898 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c95565ce3954cc598d7cbd45321d7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.760285 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet 570c7f08c46e4273bd38c32593dcf304 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:41.765337 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 4c95565ce3954cc598d7cbd45321d7c9 (DEFAULT_TABLE table=default.Table_111bf58f55b84e8088d40992e55dd3f2 [id=34dadde82105442eafc526ce3430760e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.766242 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c95565ce3954cc598d7cbd45321d7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.770725 15030 tablet_bootstrap.cc:492] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:41.771096 15030 ts_tablet_manager.cc:1403] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.211s user 0.007s sys 0.002s
I20251211 08:57:41.776558 14422 tablet_service.cc:1505] Processing CreateTablet for tablet 4c95565ce3954cc598d7cbd45321d7c9 (DEFAULT_TABLE table=default.Table_111bf58f55b84e8088d40992e55dd3f2 [id=34dadde82105442eafc526ce3430760e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.780983 14277 tablet_service.cc:1505] Processing CreateTablet for tablet f56fd93aeac6435ea84388ce05580113 (DEFAULT_TABLE table=default.Table_b997f016aa90456c8c6b1a4dcf5f4c75 [id=dece67928a044390a750633247cfc2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.781904 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f56fd93aeac6435ea84388ce05580113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.793828 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c95565ce3954cc598d7cbd45321d7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.798399 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_765e4ba63db247d4b7c358923033b6b6" } modify_external_catalogs: true
I20251211 08:57:41.798949 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_765e4ba63db247d4b7c358923033b6b6" } modify_external_catalogs: true
I20251211 08:57:41.806173 14277 tablet_service.cc:1505] Processing CreateTablet for tablet f37556f36f7e4001b03f44e545a2ba7f (DEFAULT_TABLE table=default.Table_bd6d3c8d04194e688e953991eb51e39a [id=80ce72e0c13d45719e95f0f2dc080ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.807277 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f37556f36f7e4001b03f44e545a2ba7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.811385 14420 tablet_service.cc:1505] Processing CreateTablet for tablet f37556f36f7e4001b03f44e545a2ba7f (DEFAULT_TABLE table=default.Table_bd6d3c8d04194e688e953991eb51e39a [id=80ce72e0c13d45719e95f0f2dc080ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.812495 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f37556f36f7e4001b03f44e545a2ba7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.814381 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 061e2b441d9c4b03b12d8ed31dce9595 (DEFAULT_TABLE table=default.Table_ed34b7fd8ce54c44a953a28d2fe518db [id=5c672df5eaea45419524960037727bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.811578 14426 tablet_service.cc:1505] Processing CreateTablet for tablet f56fd93aeac6435ea84388ce05580113 (DEFAULT_TABLE table=default.Table_b997f016aa90456c8c6b1a4dcf5f4c75 [id=dece67928a044390a750633247cfc2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.815330 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061e2b441d9c4b03b12d8ed31dce9595. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.815449 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f56fd93aeac6435ea84388ce05580113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.820914 14585 tablet_service.cc:1505] Processing CreateTablet for tablet f56fd93aeac6435ea84388ce05580113 (DEFAULT_TABLE table=default.Table_b997f016aa90456c8c6b1a4dcf5f4c75 [id=dece67928a044390a750633247cfc2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.820935 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 061e2b441d9c4b03b12d8ed31dce9595 (DEFAULT_TABLE table=default.Table_ed34b7fd8ce54c44a953a28d2fe518db [id=5c672df5eaea45419524960037727bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.821861 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f56fd93aeac6435ea84388ce05580113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.822345 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061e2b441d9c4b03b12d8ed31dce9595. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.804857 15318 raft_consensus.cc:493] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:41.823086 15318 raft_consensus.cc:515] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.824647 15318 leader_election.cc:290] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:41.825064 15318 raft_consensus.cc:493] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:41.825410 15318 raft_consensus.cc:515] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:41.827870 14004 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88860fa9d7e6414185f0a1cbc78b2571 consensus state has no leader
W20251211 08:57:41.831730 14542 leader_election.cc:343] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 8ad600554c0946e0bcb0da6a42be2ffd
I20251211 08:57:41.832260 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_3470f7a0edc3474884b6f95482ed49c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:41.833637 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3470f7a0edc3474884b6f95482ed49c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:41.839658 14586 tablet_service.cc:1505] Processing CreateTablet for tablet f37556f36f7e4001b03f44e545a2ba7f (DEFAULT_TABLE table=default.Table_bd6d3c8d04194e688e953991eb51e39a [id=80ce72e0c13d45719e95f0f2dc080ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.840684 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f37556f36f7e4001b03f44e545a2ba7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.842540 15318 leader_election.cc:290] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:41.842700 14905 ts_tablet_manager.cc:1929] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:41.843101 14905 log.cc:1199] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/3c5315f708c04322919c714dc915a483
I20251211 08:57:41.843664 14905 ts_tablet_manager.cc:1950] T 3c5315f708c04322919c714dc915a483 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:41.844637 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e961c5a3b5f640cbaea78b6185048d04" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:41.845204 14449 raft_consensus.cc:2468] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:41.847213 14542 leader_election.cc:304] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:41.847983 15318 raft_consensus.cc:2804] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:41.848342 15318 raft_consensus.cc:493] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:41.848695 15318 raft_consensus.cc:3060] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:41.669371 14722 raft_consensus.cc:359] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.851197 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 143f20f5c37948068ff2fb0d92f0f34b (DEFAULT_TABLE table=default.Table_3ec948949bdb4957b702fc9f5b18207a [id=9eb6a42c3a1e48f69acc0948e22afb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.851424 14722 raft_consensus.cc:385] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:41.851646 14722 raft_consensus.cc:740] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:41.854199 15030 raft_consensus.cc:359] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.854821 15030 raft_consensus.cc:385] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:41.855086 15030 raft_consensus.cc:740] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:41.855288 15318 raft_consensus.cc:515] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.855628 15030 consensus_queue.cc:260] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.857491 15030 ts_tablet_manager.cc:1434] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.086s user 0.002s sys 0.002s
I20251211 08:57:41.858106 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e961c5a3b5f640cbaea78b6185048d04" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:41.858605 14449 raft_consensus.cc:3060] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:41.694628 15299 rpcz_store.cc:276] 1211 08:57:40.534428 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.535320 (+ 892us) service_pool.cc:225] Handling call
1211 08:57:41.694111 (+1158791us) ts_tablet_manager.cc:1151] Deleting tablet 49015c5d4f2b4d07b547a9df57c992eb
1211 08:57:41.694141 (+ 30us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.694343 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1157324}
I20251211 08:57:41.859900 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 271) took 1241 ms. Trace:
I20251211 08:57:41.860141 15299 rpcz_store.cc:276] 1211 08:57:40.618217 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.620459 (+ 2242us) service_pool.cc:225] Handling call
1211 08:57:41.859621 (+1239162us) ts_tablet_manager.cc:1151] Deleting tablet f5b462cf21e3420c8e323a4ac4f04abe
1211 08:57:41.859657 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.859856 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1180962}
I20251211 08:57:41.861528 14714 tablet_bootstrap.cc:654] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:41.862718 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 49015c5d4f2b4d07b547a9df57c992eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49015c5d4f2b4d07b547a9df57c992eb already in progress: creating tablet
W20251211 08:57:41.863365 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f5b462cf21e3420c8e323a4ac4f04abe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5b462cf21e3420c8e323a4ac4f04abe already in progress: creating tablet
I20251211 08:57:41.864187 14722 consensus_queue.cc:260] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.866218 14722 ts_tablet_manager.cc:1434] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.199s user 0.003s sys 0.001s
W20251211 08:57:41.874331 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0bb339322504704ae57bba2f4c3a9bb has no consensus state
I20251211 08:57:41.875976 14904 ts_tablet_manager.cc:1929] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:41.876309 14904 log.cc:1199] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/3c5315f708c04322919c714dc915a483
I20251211 08:57:41.876945 14904 ts_tablet_manager.cc:1950] T 3c5315f708c04322919c714dc915a483 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:41.877393 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 272) took 1170 ms. Trace:
I20251211 08:57:41.877697 15299 rpcz_store.cc:276] 1211 08:57:40.707074 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.707203 (+ 129us) service_pool.cc:225] Handling call
1211 08:57:41.860820 (+1153617us) ts_tablet_manager.cc:1151] Deleting tablet 1c02cb6ec01041dba118683a57aecde2
1211 08:57:41.860856 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.877328 (+ 16472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1153118}
I20251211 08:57:41.878646 15299 tablet_replica.cc:333] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
W20251211 08:57:41.879504 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 1c02cb6ec01041dba118683a57aecde2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c02cb6ec01041dba118683a57aecde2 already in progress: creating tablet
I20251211 08:57:41.880654 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ad600554c0946e0bcb0da6a42be2ffd" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:41.880600 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4e15a6b7343e46c491019358d60e3589"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:41.880936 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e961c5a3b5f640cbaea78b6185048d04" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:41.882009 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e15a6b7343e46c491019358d60e3589 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:41.882740 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e961c5a3b5f640cbaea78b6185048d04" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:41.886149 14277 tablet_service.cc:1505] Processing CreateTablet for tablet c03f976b926a4a6fba8b70e9df9a4160 (DEFAULT_TABLE table=default.Table_3867d5bf46e94dd1990e38dd47037918 [id=ce2647a00ea24733bc2fa5f6cb36ffec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.886688 14722 tablet_bootstrap.cc:492] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:41.887079 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03f976b926a4a6fba8b70e9df9a4160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.889340 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 061e2b441d9c4b03b12d8ed31dce9595 (DEFAULT_TABLE table=default.Table_ed34b7fd8ce54c44a953a28d2fe518db [id=5c672df5eaea45419524960037727bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.890260 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061e2b441d9c4b03b12d8ed31dce9595. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.894055 14722 tablet_bootstrap.cc:654] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:41.898542 14714 tablet_bootstrap.cc:492] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:41.898972 14714 ts_tablet_manager.cc:1403] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.220s user 0.011s sys 0.000s
I20251211 08:57:41.901079 14714 raft_consensus.cc:359] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.901660 14714 raft_consensus.cc:385] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:41.901877 14714 raft_consensus.cc:740] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:41.902757 14449 raft_consensus.cc:2468] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:41.903705 15299 raft_consensus.cc:2243] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:41.904218 15299 raft_consensus.cc:2272] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:41.904753 14542 leader_election.cc:304] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:41.905465 15428 raft_consensus.cc:2804] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:41.905865 15428 raft_consensus.cc:697] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:41.906554 15428 consensus_queue.cc:237] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.910720 14585 tablet_service.cc:1505] Processing CreateTablet for tablet c03f976b926a4a6fba8b70e9df9a4160 (DEFAULT_TABLE table=default.Table_3867d5bf46e94dd1990e38dd47037918 [id=ce2647a00ea24733bc2fa5f6cb36ffec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.911960 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03f976b926a4a6fba8b70e9df9a4160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.911217 15318 leader_election.cc:290] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:41.918148 14422 tablet_service.cc:1505] Processing CreateTablet for tablet c03f976b926a4a6fba8b70e9df9a4160 (DEFAULT_TABLE table=default.Table_3867d5bf46e94dd1990e38dd47037918 [id=ce2647a00ea24733bc2fa5f6cb36ffec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:41.919106 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03f976b926a4a6fba8b70e9df9a4160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:41.897887 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 242) took 1327 ms. Trace:
I20251211 08:57:41.925556 14904 rpcz_store.cc:276] 1211 08:57:40.570347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.570458 (+ 111us) service_pool.cc:225] Handling call
1211 08:57:41.422701 (+852243us) ts_tablet_manager.cc:1151] Deleting tablet 3c5315f708c04322919c714dc915a483
1211 08:57:41.422740 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.629946 (+207206us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:41.875765 (+245819us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:41.897830 (+ 22065us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19079,"tablet-delete.queue_time_us":732062}
I20251211 08:57:41.926739 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 3c5315f708c04322919c714dc915a483 (table default.Table_d12516554d1547419ff004d016cb9a3a [id=1078a3f3dca34fbe8d48d184f434c7b6]) successfully deleted
I20251211 08:57:41.930130 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 243) took 1311 ms. Trace:
I20251211 08:57:41.930583 15030 tablet_bootstrap.cc:492] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:41.929175 15299 ts_tablet_manager.cc:1916] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:41.931758 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 223) took 2283 ms. Trace:
I20251211 08:57:41.932098 14905 rpcz_store.cc:276] 1211 08:57:39.648042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.648140 (+ 98us) service_pool.cc:225] Handling call
1211 08:57:41.191202 (+1543062us) ts_tablet_manager.cc:1151] Deleting tablet 3c5315f708c04322919c714dc915a483
1211 08:57:41.191247 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.453908 (+262661us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:41.842517 (+388609us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:41.931705 (+ 89188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":183,"spinlock_wait_cycles":3968,"tablet-delete.queue_time_us":1542480}
I20251211 08:57:41.933415 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 224) took 2285 ms. Trace:
I20251211 08:57:41.933712 14905 rpcz_store.cc:276] 1211 08:57:39.648365 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.648485 (+ 120us) service_pool.cc:225] Handling call
1211 08:57:41.933183 (+2284698us) ts_tablet_manager.cc:1151] Deleting tablet ad903727f79a4180adf047c92d981c7e
1211 08:57:41.933225 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.933368 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2284226}
I20251211 08:57:41.934723 14905 tablet_replica.cc:333] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:41.934651 15360 raft_consensus.cc:515] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.935652 14905 raft_consensus.cc:2243] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:41.935928 15030 tablet_bootstrap.cc:654] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:41.936664 14905 raft_consensus.cc:2272] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:41.936977 14904 rpcz_store.cc:276] 1211 08:57:40.619181 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.619300 (+ 119us) service_pool.cc:225] Handling call
1211 08:57:41.929833 (+1310533us) ts_tablet_manager.cc:1151] Deleting tablet f5b462cf21e3420c8e323a4ac4f04abe
1211 08:57:41.929871 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.930079 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1239026}
W20251211 08:57:41.938083 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet f5b462cf21e3420c8e323a4ac4f04abe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5b462cf21e3420c8e323a4ac4f04abe already in progress: creating tablet
I20251211 08:57:41.938817 14905 ts_tablet_manager.cc:1916] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:41.944082 15030 tablet_bootstrap.cc:492] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:41.944432 15030 ts_tablet_manager.cc:1403] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20251211 08:57:41.953091 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 244) took 1248 ms. Trace:
I20251211 08:57:41.953398 14904 rpcz_store.cc:276] 1211 08:57:40.704976 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.705208 (+ 232us) service_pool.cc:225] Handling call
1211 08:57:41.952753 (+1247545us) ts_tablet_manager.cc:1151] Deleting tablet 1c02cb6ec01041dba118683a57aecde2
1211 08:57:41.952792 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.953030 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1247067}
I20251211 08:57:41.902369 14714 consensus_queue.cc:260] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.957592 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_bbfc3f82c89f4bfab0a97bd49febabd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:41.959230 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbfc3f82c89f4bfab0a97bd49febabd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:41.962284 14905 ts_tablet_manager.cc:1929] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:41.962649 14905 log.cc:1199] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/42720f5f5bf445f2b72e513ad4a6d95f
I20251211 08:57:41.963353 14905 ts_tablet_manager.cc:1950] T 42720f5f5bf445f2b72e513ad4a6d95f P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:41.965066 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 225) took 2313 ms. Trace:
I20251211 08:57:41.965410 14905 rpcz_store.cc:276] 1211 08:57:39.651526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.651631 (+ 105us) service_pool.cc:225] Handling call
1211 08:57:41.934596 (+2282965us) ts_tablet_manager.cc:1151] Deleting tablet 42720f5f5bf445f2b72e513ad4a6d95f
1211 08:57:41.934628 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.949191 (+ 14563us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:41.962085 (+ 12894us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:41.965014 (+ 2929us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2282465}
I20251211 08:57:41.966645 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 226) took 2294 ms. Trace:
I20251211 08:57:41.966969 14905 rpcz_store.cc:276] 1211 08:57:39.671895 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.672121 (+ 226us) service_pool.cc:225] Handling call
1211 08:57:41.966346 (+2294225us) ts_tablet_manager.cc:1151] Deleting tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0
1211 08:57:41.966399 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.966602 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2293720}
W20251211 08:57:41.967492 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 1c02cb6ec01041dba118683a57aecde2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c02cb6ec01041dba118683a57aecde2 already in progress: creating tablet
I20251211 08:57:41.968075 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 227) took 2255 ms. Trace:
I20251211 08:57:41.968295 14904 tablet_replica.cc:333] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:41.968389 14905 rpcz_store.cc:276] 1211 08:57:39.712700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.772105 (+ 59405us) service_pool.cc:225] Handling call
1211 08:57:41.967741 (+2195636us) ts_tablet_manager.cc:1151] Deleting tablet da57b72207414512b9744b283bc869fe
1211 08:57:41.967836 (+ 95us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.968032 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2195023}
I20251211 08:57:41.969026 14904 raft_consensus.cc:2243] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:41.969530 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 237) took 1813 ms. Trace:
I20251211 08:57:41.969800 14904 raft_consensus.cc:2272] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:41.969830 14905 rpcz_store.cc:276] 1211 08:57:40.155666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.158815 (+ 3149us) service_pool.cc:225] Handling call
1211 08:57:41.969214 (+1810399us) ts_tablet_manager.cc:1151] Deleting tablet 4e813a4c73334c839bfe6af2119e259c
1211 08:57:41.969298 (+ 84us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.969492 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1809844}
I20251211 08:57:41.970891 14905 tablet_replica.cc:333] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:41.971362 14714 ts_tablet_manager.cc:1434] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.072s user 0.006s sys 0.000s
I20251211 08:57:41.971625 14905 raft_consensus.cc:2243] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:41.972385 14905 pending_rounds.cc:70] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:57:41.972625 14905 pending_rounds.cc:77] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256408512118784 op_type: NO_OP noop_request { }
I20251211 08:57:41.972978 14905 raft_consensus.cc:2889] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:41.973337 14905 raft_consensus.cc:2272] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:41.975009 14905 ts_tablet_manager.cc:1916] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:41.938169 15360 leader_election.cc:290] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:41.946532 15030 raft_consensus.cc:359] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.978824 15030 raft_consensus.cc:385] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:41.979092 15030 raft_consensus.cc:740] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:41.980093 15030 consensus_queue.cc:260] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:41.982012 15030 ts_tablet_manager.cc:1434] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.037s user 0.006s sys 0.000s
I20251211 08:57:41.982749 15030 tablet_bootstrap.cc:492] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:41.987756 14714 tablet_bootstrap.cc:492] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:41.990478 15030 tablet_bootstrap.cc:654] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:41.999361 14904 ts_tablet_manager.cc:1916] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.001590 15030 tablet_bootstrap.cc:492] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:42.001978 15030 ts_tablet_manager.cc:1403] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.006s
W20251211 08:57:42.027168 13999 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 -> Peer 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015): Couldn't send request to peer 73702d54989f4959acfc4a6fdcede6ca. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20251211 08:57:42.027817 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 3c5315f708c04322919c714dc915a483 (table default.Table_d12516554d1547419ff004d016cb9a3a [id=1078a3f3dca34fbe8d48d184f434c7b6]) successfully deleted
W20251211 08:57:42.028463 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet ad903727f79a4180adf047c92d981c7e because the tablet was not found. No further retry: Not found: Tablet not found: ad903727f79a4180adf047c92d981c7e
I20251211 08:57:42.029474 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 42720f5f5bf445f2b72e513ad4a6d95f (table default.Table_58593e18b9634b2c8912ff1109e05a52 [id=8bb6612dda374661a7d3a8a06bd230ae]) successfully deleted
W20251211 08:57:42.029984 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 already in progress: creating tablet
I20251211 08:57:42.032362 14722 tablet_bootstrap.cc:492] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:42.032805 14722 ts_tablet_manager.cc:1403] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.146s user 0.004s sys 0.006s
I20251211 08:57:42.038388 14010 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 2a1b011cb7c647f79bb16ce1c92bcff5
I20251211 08:57:42.040727 14010 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_fd201f899f064ec0b339bd8b7679fefc" } new_table_name: "default.Table_321194b0f71d457e81e14c810650b486" modify_external_catalogs: true
W20251211 08:57:42.030540 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet da57b72207414512b9744b283bc869fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da57b72207414512b9744b283bc869fe already in progress: creating tablet
W20251211 08:57:42.041693 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 4e813a4c73334c839bfe6af2119e259c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e813a4c73334c839bfe6af2119e259c already in progress: creating tablet
I20251211 08:57:42.041886 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.042277 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:42.044258 14722 raft_consensus.cc:359] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.047583 14722 raft_consensus.cc:385] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:42.047952 14722 raft_consensus.cc:740] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:42.048713 14722 consensus_queue.cc:260] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.053061 14714 tablet_bootstrap.cc:654] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:42.060818 14722 ts_tablet_manager.cc:1434] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20251211 08:57:42.062956 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dca342fd9de241b5884600199f06fbeb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:42.063483 14449 raft_consensus.cc:3060] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.065055 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 570c7f08c46e4273bd38c32593dcf304
I20251211 08:57:42.066263 14013 catalog_manager.cc:6724] ReplaceTablet: tablet 570c7f08c46e4273bd38c32593dcf304 deleted and replaced by tablet 3838fd220d034b3595f351b906188339
I20251211 08:57:42.069854 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 78c6a703e1e44f41931fe465def01bf6
I20251211 08:57:42.069876 14422 tablet_service.cc:1552] Processing DeleteTablet for tablet 570c7f08c46e4273bd38c32593dcf304 with delete_type TABLET_DATA_DELETED (replaced by tablet 3838fd220d034b3595f351b906188339) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.071345 14449 raft_consensus.cc:2468] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:42.072623 14542 leader_election.cc:304] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:42.073637 14422 tablet_service.cc:1552] Processing DeleteTablet for tablet 78c6a703e1e44f41931fe465def01bf6 with delete_type TABLET_DATA_DELETED (replaced by tablet 02c7fbd61b2e42efaa0ae06ed1b1091c) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.073958 14012 catalog_manager.cc:6724] ReplaceTablet: tablet 78c6a703e1e44f41931fe465def01bf6 deleted and replaced by tablet 02c7fbd61b2e42efaa0ae06ed1b1091c
I20251211 08:57:42.076157 14009 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_fd2daf1c25534c25b34cf363453b0e67 [id=6e2d344612214ee382863680435345b7] alter complete (version 1)
I20251211 08:57:42.085564 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 78c6a703e1e44f41931fe465def01bf6 with delete_type TABLET_DATA_DELETED (replaced by tablet 02c7fbd61b2e42efaa0ae06ed1b1091c) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:42.085882 14544 leader_election.cc:343] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.087186 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.088307 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 570c7f08c46e4273bd38c32593dcf304 with delete_type TABLET_DATA_DELETED (replaced by tablet 3838fd220d034b3595f351b906188339) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.089126 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143f20f5c37948068ff2fb0d92f0f34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.092079 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dca342fd9de241b5884600199f06fbeb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:42.068377 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet 570c7f08c46e4273bd38c32593dcf304 with delete_type TABLET_DATA_DELETED (replaced by tablet 3838fd220d034b3595f351b906188339) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:42.075415 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 78c6a703e1e44f41931fe465def01bf6 with delete_type TABLET_DATA_DELETED (replaced by tablet 02c7fbd61b2e42efaa0ae06ed1b1091c) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:42.079041 15368 raft_consensus.cc:493] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:42.093308 15368 raft_consensus.cc:515] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.099819 15368 leader_election.cc:290] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:42.101716 14231 leader_election.cc:343] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: db302282bc3d4d8cb56d1cbb2f73c4fe
I20251211 08:57:42.086680 14544 leader_election.cc:304] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
W20251211 08:57:42.103839 14544 leader_election.cc:343] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:42.115581 14544 leader_election.cc:343] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.116138 14905 ts_tablet_manager.cc:1929] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:42.116750 14905 log.cc:1199] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/5daba165262e41af8d3481d3b4929092
I20251211 08:57:42.119619 14905 ts_tablet_manager.cc:1950] T 5daba165262e41af8d3481d3b4929092 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:42.122303 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 242) took 1694 ms. Trace:
I20251211 08:57:42.122413 14296 raft_consensus.cc:3060] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.133940 15299 ts_tablet_manager.cc:1929] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:42.134274 15299 log.cc:1199] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/4e410da32b3446edbf84b2eaa96424c3
I20251211 08:57:42.138877 14005 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 14957cc3ce5745f28d6e7e269fcf6d3a
I20251211 08:57:42.139784 15030 raft_consensus.cc:359] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.140748 15030 raft_consensus.cc:385] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:42.140941 15318 raft_consensus.cc:2804] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:42.141041 15030 raft_consensus.cc:740] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:42.141429 15318 raft_consensus.cc:697] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:42.141634 15030 consensus_queue.cc:260] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.142246 15318 consensus_queue.cc:237] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.148337 15428 raft_consensus.cc:2749] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:42.156517 15299 ts_tablet_manager.cc:1950] T 4e410da32b3446edbf84b2eaa96424c3 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:42.160108 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 273) took 1178 ms. Trace:
I20251211 08:57:42.160405 15299 rpcz_store.cc:276] 1211 08:57:40.982019 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.983219 (+ 1200us) service_pool.cc:225] Handling call
1211 08:57:41.878516 (+895297us) ts_tablet_manager.cc:1151] Deleting tablet 4e410da32b3446edbf84b2eaa96424c3
1211 08:57:41.878552 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.013630 (+135078us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.133731 (+120101us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.160059 (+ 26328us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24141,"tablet-delete.queue_time_us":894795}
I20251211 08:57:42.161509 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 4e410da32b3446edbf84b2eaa96424c3 (table default.Table_eb447b641c624aeba08c478e373a532d [id=38f5e9afc13a4797a7e8460f1f073280]) successfully deleted
I20251211 08:57:42.163938 14722 tablet_bootstrap.cc:492] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:42.174072 15299 tablet_replica.cc:333] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:42.174574 15030 ts_tablet_manager.cc:1434] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.172s user 0.009s sys 0.004s
I20251211 08:57:42.175639 14714 tablet_bootstrap.cc:492] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:42.176224 15030 tablet_bootstrap.cc:492] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:42.176918 14714 ts_tablet_manager.cc:1403] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.189s user 0.006s sys 0.008s
I20251211 08:57:42.177289 14904 ts_tablet_manager.cc:1929] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:42.177814 14904 log.cc:1199] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/4e410da32b3446edbf84b2eaa96424c3
I20251211 08:57:42.183367 15299 raft_consensus.cc:2243] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:42.183840 15299 raft_consensus.cc:2272] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:42.184695 14714 raft_consensus.cc:359] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.185482 14714 raft_consensus.cc:385] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:42.185781 14714 raft_consensus.cc:740] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:42.186012 15299 ts_tablet_manager.cc:1916] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.187732 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e83a6dbb039a4f789964450c022037e6" } new_table_name: "default.Table_7cf998dd4bd84149b73b791cff8c3652" modify_external_catalogs: true
I20251211 08:57:42.192713 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_df3433d6d1fb4af39546e642816b6331"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:42.194135 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df3433d6d1fb4af39546e642816b6331 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:42.203938 15030 tablet_bootstrap.cc:654] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:42.122628 14905 rpcz_store.cc:276] 1211 08:57:40.427609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.427738 (+ 129us) service_pool.cc:225] Handling call
1211 08:57:41.970763 (+1543025us) ts_tablet_manager.cc:1151] Deleting tablet 5daba165262e41af8d3481d3b4929092
1211 08:57:41.970815 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:41.983124 (+ 12309us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.115698 (+132574us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.122249 (+ 6551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542515}
I20251211 08:57:42.213881 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 243) took 1709 ms. Trace:
I20251211 08:57:42.213840 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 5daba165262e41af8d3481d3b4929092 (table default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]) successfully deleted
I20251211 08:57:42.214184 14905 rpcz_store.cc:276] 1211 08:57:40.504060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.504178 (+ 118us) service_pool.cc:225] Handling call
1211 08:57:42.213576 (+1709398us) ts_tablet_manager.cc:1151] Deleting tablet afe339c4cc95490683618e18a3da252b
1211 08:57:42.213638 (+ 62us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.213835 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1697469}
I20251211 08:57:42.215523 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 244) took 1680 ms. Trace:
I20251211 08:57:42.215797 14905 rpcz_store.cc:276] 1211 08:57:40.534939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.535066 (+ 127us) service_pool.cc:225] Handling call
1211 08:57:42.215167 (+1680101us) ts_tablet_manager.cc:1151] Deleting tablet 49015c5d4f2b4d07b547a9df57c992eb
1211 08:57:42.215222 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.215480 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1679662}
W20251211 08:57:42.216342 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet afe339c4cc95490683618e18a3da252b because the tablet was not found. No further retry: Not found: Tablet not found: afe339c4cc95490683618e18a3da252b
I20251211 08:57:42.216794 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 245) took 1600 ms. Trace:
W20251211 08:57:42.217175 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 49015c5d4f2b4d07b547a9df57c992eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49015c5d4f2b4d07b547a9df57c992eb already in progress: creating tablet
I20251211 08:57:42.217018 14905 rpcz_store.cc:276] 1211 08:57:40.616525 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.616660 (+ 135us) service_pool.cc:225] Handling call
1211 08:57:42.216505 (+1599845us) ts_tablet_manager.cc:1151] Deleting tablet f5b462cf21e3420c8e323a4ac4f04abe
1211 08:57:42.216551 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.216756 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599265}
W20251211 08:57:42.217901 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet f5b462cf21e3420c8e323a4ac4f04abe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5b462cf21e3420c8e323a4ac4f04abe already in progress: creating tablet
W20251211 08:57:42.224329 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 107496us
W20251211 08:57:42.226022 14544 leader_election.cc:343] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.226095 15428 raft_consensus.cc:493] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:42.186539 14714 consensus_queue.cc:260] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.226516 15428 raft_consensus.cc:515] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:42.227170 14010 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 consensus state has no leader
I20251211 08:57:42.228196 15428 leader_election.cc:290] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:42.228436 15440 raft_consensus.cc:493] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:42.228842 15440 raft_consensus.cc:515] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.233318 15452 tablet.cc:1722] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:42.242532 14714 ts_tablet_manager.cc:1434] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.065s user 0.000s sys 0.005s
I20251211 08:57:42.244128 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e444843f69064fbb9c2a6d0671fd9a2a" } modify_external_catalogs: true
I20251211 08:57:42.244622 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e444843f69064fbb9c2a6d0671fd9a2a" } modify_external_catalogs: true
I20251211 08:57:42.247457 14011 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_b19d002dfb154df6aadac72b8859cce9 [id=0944cc290d954cf4a33fec11acb53315] alter complete (version 1)
I20251211 08:57:42.249274 14904 ts_tablet_manager.cc:1950] T 4e410da32b3446edbf84b2eaa96424c3 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:42.251274 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 245) took 1247 ms. Trace:
I20251211 08:57:42.251572 14904 rpcz_store.cc:276] 1211 08:57:41.003849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.003960 (+ 111us) service_pool.cc:225] Handling call
1211 08:57:41.968162 (+964202us) ts_tablet_manager.cc:1151] Deleting tablet 4e410da32b3446edbf84b2eaa96424c3
1211 08:57:41.968200 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.140176 (+171976us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.177043 (+ 36867us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.251219 (+ 74176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":963721}
I20251211 08:57:42.252807 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 4e410da32b3446edbf84b2eaa96424c3 (table default.Table_eb447b641c624aeba08c478e373a532d [id=38f5e9afc13a4797a7e8460f1f073280]) successfully deleted
I20251211 08:57:42.256443 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c53035e83710403bac3a2b401c8c97ad" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:42.256989 14448 raft_consensus.cc:2468] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:42.257807 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b5a4b6b1b28406c82e2acd9356b60d4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:42.257997 14542 leader_election.cc:304] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:42.258359 14445 raft_consensus.cc:2468] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:42.259459 15443 raft_consensus.cc:2804] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:42.259757 15443 raft_consensus.cc:493] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:42.260058 15443 raft_consensus.cc:3060] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.263298 14542 leader_election.cc:304] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:42.263934 15428 raft_consensus.cc:2804] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:42.264225 15428 raft_consensus.cc:493] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:42.264518 15428 raft_consensus.cc:3060] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.267589 14714 tablet_bootstrap.cc:492] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:42.267887 14722 tablet_bootstrap.cc:654] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:42.275945 15443 raft_consensus.cc:515] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.276685 14904 tablet_replica.cc:333] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:42.277426 14904 raft_consensus.cc:2243] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:42.278263 14904 raft_consensus.cc:2272] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:42.278724 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c53035e83710403bac3a2b401c8c97ad" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:42.279309 14445 raft_consensus.cc:3060] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:42.285923 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 69806.4us
I20251211 08:57:42.286304 14445 raft_consensus.cc:2468] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:42.287923 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db302282bc3d4d8cb56d1cbb2f73c4fe" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:42.288537 14607 raft_consensus.cc:2468] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:42.288708 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b5a4b6b1b28406c82e2acd9356b60d4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:42.286751 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c53035e83710403bac3a2b401c8c97ad" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:42.290695 14542 leader_election.cc:304] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:42.291172 14904 ts_tablet_manager.cc:1916] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.259467 15440 leader_election.cc:290] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:42.291726 15440 raft_consensus.cc:2804] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:42.292644 15440 raft_consensus.cc:697] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:42.292627 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c53035e83710403bac3a2b401c8c97ad" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:42.293315 15440 consensus_queue.cc:237] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.296523 15428 raft_consensus.cc:515] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:42.299111 14544 leader_election.cc:343] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:42.299844 14544 leader_election.cc:343] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:42.300520 14544 leader_election.cc:343] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.301151 14230 leader_election.cc:304] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:42.302017 15368 raft_consensus.cc:2804] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:42.302318 15368 raft_consensus.cc:493] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:42.302598 15368 raft_consensus.cc:3060] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.303260 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b5a4b6b1b28406c82e2acd9356b60d4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:42.303772 14448 raft_consensus.cc:3060] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.304481 15428 leader_election.cc:290] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:42.310178 15368 raft_consensus.cc:515] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:42.313351 14231 leader_election.cc:343] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: db302282bc3d4d8cb56d1cbb2f73c4fe
I20251211 08:57:42.313762 15368 leader_election.cc:290] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:42.316744 15030 tablet_bootstrap.cc:492] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:42.317196 15030 ts_tablet_manager.cc:1403] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.141s user 0.008s sys 0.000s
I20251211 08:57:42.310608 14448 raft_consensus.cc:2468] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:42.327509 14009 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_0ba5761cf15c49bd81994a532279c459 [id=1a1830702e5f48b4939784eee3470127] alter complete (version 1)
W20251211 08:57:42.329035 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 99612.4us
I20251211 08:57:42.329802 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet b8d753358f20476ba23d7d7b72e26e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:42.330049 14009 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 45) took 1174 ms (client timeout 1000 ms). Trace:
I20251211 08:57:42.327018 15299 ts_tablet_manager.cc:1929] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:42.330627 15299 log.cc:1199] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/eb5a6e533e1f4da7b03a800d029f712d
I20251211 08:57:42.331491 15299 ts_tablet_manager.cc:1950] T eb5a6e533e1f4da7b03a800d029f712d P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:42.268705 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 246) took 1564 ms. Trace:
I20251211 08:57:42.331835 14905 rpcz_store.cc:276] 1211 08:57:40.704519 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:40.704629 (+ 110us) service_pool.cc:225] Handling call
1211 08:57:42.268317 (+1563688us) ts_tablet_manager.cc:1151] Deleting tablet 1c02cb6ec01041dba118683a57aecde2
1211 08:57:42.268366 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.268655 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563216}
W20251211 08:57:42.332842 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 1c02cb6ec01041dba118683a57aecde2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c02cb6ec01041dba118683a57aecde2 already in progress: creating tablet
W20251211 08:57:42.333803 14011 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 35) took 975 ms (client timeout 1000 ms). Trace:
I20251211 08:57:42.335040 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet a6570281e1064c79b6e2297723f28658 with delete_type TABLET_DATA_DELETED (replaced by tablet e9905227c0a844d199362a442e7d630c) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.336273 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 174fefd70bcc41418a77a5716f97cd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:42.334102 14011 rpcz_store.cc:269] 1211 08:57:41.358599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.358707 (+ 108us) service_pool.cc:225] Handling call
1211 08:57:42.333755 (+975048us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:42.195945 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:42.196322 (+ 377us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:42.196344 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:57:42.196355 (+ 11us) tablet.cc:660] Decoding operations
1211 08:57:42.196636 (+ 281us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:42.196679 (+ 43us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:42.196699 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:42.196829 (+ 130us) tablet.cc:699] Row locks acquired
1211 08:57:42.196840 (+ 11us) write_op.cc:260] PREPARE: finished
1211 08:57:42.196933 (+ 93us) write_op.cc:270] Start()
1211 08:57:42.197028 (+ 95us) write_op.cc:276] Timestamp: P: 1765443462197000 usec, L: 0
1211 08:57:42.197042 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:42.202373 (+ 5331us) log.cc:844] Serialized 356 byte log entry
1211 08:57:42.240539 (+ 38166us) op_driver.cc:464] REPLICATION: finished
1211 08:57:42.240810 (+ 271us) write_op.cc:301] APPLY: starting
1211 08:57:42.240854 (+ 44us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:42.240936 (+ 82us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:42.241026 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:42.241174 (+ 148us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:42.241284 (+ 110us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:42.241331 (+ 47us) write_op.cc:312] APPLY: finished
1211 08:57:42.241474 (+ 143us) log.cc:844] Serialized 23 byte log entry
1211 08:57:42.247103 (+ 5629us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:42.247161 (+ 58us) write_op.cc:454] Released schema lock
1211 08:57:42.247191 (+ 30us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:42.248006 (+ 815us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:42.248040 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":151,"apply.run_cpu_time_us":1115,"apply.run_wall_time_us":7149,"num_ops":1,"prepare.queue_time_us":115775,"prepare.run_cpu_time_us":6060,"prepare.run_wall_time_us":44032,"raft.queue_time_us":19636,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":9,"replication_time_us":43372,"spinlock_wait_cycles":4352,"thread_start_us":36485,"threads_started":1}]]}
I20251211 08:57:42.337817 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet e961c5a3b5f640cbaea78b6185048d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.338660 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 274) took 1290 ms. Trace:
I20251211 08:57:42.338972 15299 rpcz_store.cc:276] 1211 08:57:41.048544 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.049140 (+ 596us) service_pool.cc:225] Handling call
1211 08:57:42.173935 (+1124795us) ts_tablet_manager.cc:1151] Deleting tablet eb5a6e533e1f4da7b03a800d029f712d
1211 08:57:42.173972 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.287852 (+113880us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.319519 (+ 31667us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.338606 (+ 19087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124214}
I20251211 08:57:42.339859 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet eb5a6e533e1f4da7b03a800d029f712d (table default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85]) successfully deleted
I20251211 08:57:42.340013 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 278) took 1175 ms. Trace:
I20251211 08:57:42.339373 14905 tablet_replica.cc:333] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:42.340732 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b5a4b6b1b28406c82e2acd9356b60d4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:42.341732 14905 raft_consensus.cc:2243] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:42.343567 15030 raft_consensus.cc:359] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.344195 15030 raft_consensus.cc:385] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:42.344424 15030 raft_consensus.cc:740] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
W20251211 08:57:42.330325 14009 rpcz_store.cc:269] 1211 08:57:41.155410 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.159325 (+ 3915us) service_pool.cc:225] Handling call
1211 08:57:41.254094 (+ 94769us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:42.329980 (+1075886us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:41.647953 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:41.648333 (+ 380us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:41.648356 (+ 23us) write_op.cc:435] Acquired schema lock
1211 08:57:41.648369 (+ 13us) tablet.cc:660] Decoding operations
1211 08:57:41.648601 (+ 232us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:41.648635 (+ 34us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:41.648656 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:41.648782 (+ 126us) tablet.cc:699] Row locks acquired
1211 08:57:41.648795 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:41.648891 (+ 96us) write_op.cc:270] Start()
1211 08:57:41.648966 (+ 75us) write_op.cc:276] Timestamp: P: 1765443461648942 usec, L: 0
1211 08:57:41.648991 (+ 25us) op_driver.cc:348] REPLICATION: starting
1211 08:57:41.649384 (+ 393us) log.cc:844] Serialized 356 byte log entry
1211 08:57:41.793568 (+144184us) op_driver.cc:464] REPLICATION: finished
1211 08:57:41.795211 (+ 1643us) write_op.cc:301] APPLY: starting
1211 08:57:41.795257 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:41.795332 (+ 75us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:41.795415 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:41.795549 (+ 134us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:41.795660 (+ 111us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:41.795708 (+ 48us) write_op.cc:312] APPLY: finished
1211 08:57:41.795884 (+ 176us) log.cc:844] Serialized 23 byte log entry
1211 08:57:41.796008 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:41.796060 (+ 52us) write_op.cc:454] Released schema lock
1211 08:57:41.796086 (+ 26us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:41.797845 (+ 1759us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:41.797870 (+ 25us) write_op.cc:454] Released schema lock
Related trace 'op':
1211 08:57:42.240098 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:42.240489 (+ 391us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:42.240514 (+ 25us) write_op.cc:435] Acquired schema lock
1211 08:57:42.240526 (+ 12us) tablet.cc:660] Decoding operations
1211 08:57:42.240764 (+ 238us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:42.240800 (+ 36us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:42.240820 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:42.240948 (+ 128us) tablet.cc:699] Row locks acquired
1211 08:57:42.240961 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:42.241062 (+ 101us) write_op.cc:270] Start()
1211 08:57:42.241132 (+ 70us) write_op.cc:276] Timestamp: P: 1765443462241149 usec, L: 0
1211 08:57:42.241145 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:42.241580 (+ 435us) log.cc:844] Serialized 356 byte log entry
1211 08:57:42.326142 (+ 84562us) op_driver.cc:464] REPLICATION: finished
1211 08:57:42.326349 (+ 207us) write_op.cc:301] APPLY: starting
1211 08:57:42.326403 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:42.326478 (+ 75us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:42.326561 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:42.326707 (+ 146us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:42.326839 (+ 132us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:42.326876 (+ 37us) write_op.cc:312] APPLY: finished
1211 08:57:42.327037 (+ 161us) log.cc:844] Serialized 23 byte log entry
1211 08:57:42.327174 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:42.327220 (+ 46us) write_op.cc:454] Released schema lock
1211 08:57:42.327246 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":66203,"reported_tablets":4,"spinlock_wait_cycles":72197376,"child_traces":[["op",{"apply.queue_time_us":1494,"apply.run_cpu_time_us":1107,"apply.run_wall_time_us":2597,"num_ops":1,"prepare.queue_time_us":320630,"prepare.run_cpu_time_us":1619,"prepare.run_wall_time_us":10918,"replication_time_us":144469}],["op",{"apply.queue_time_us":71,"apply.run_cpu_time_us":1077,"apply.run_wall_time_us":12572,"num_ops":1,"prepare.queue_time_us":136772,"prepare.run_cpu_time_us":1638,"prepare.run_wall_time_us":1637,"replication_time_us":84907}]]}
I20251211 08:57:42.345783 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db302282bc3d4d8cb56d1cbb2f73c4fe" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:42.346328 14604 raft_consensus.cc:3060] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:42.346385 14544 leader_election.cc:343] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:42.347059 14544 consensus_peers.cc:597] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:42.340250 15299 rpcz_store.cc:276] 1211 08:57:41.165017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.165144 (+ 127us) service_pool.cc:225] Handling call
1211 08:57:42.339728 (+1174584us) ts_tablet_manager.cc:1151] Deleting tablet 080d1700df4a4df09acda65c6d021d96
1211 08:57:42.339763 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.339971 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1174041}
W20251211 08:57:42.352774 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 080d1700df4a4df09acda65c6d021d96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 080d1700df4a4df09acda65c6d021d96 already in progress: creating tablet
I20251211 08:57:42.355684 14904 ts_tablet_manager.cc:1929] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:42.356032 14904 log.cc:1199] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/eb5a6e533e1f4da7b03a800d029f712d
I20251211 08:57:42.351683 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 284) took 1013 ms. Trace:
I20251211 08:57:42.356568 15299 rpcz_store.cc:276] 1211 08:57:41.338643 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.356075 (+ 17432us) service_pool.cc:225] Handling call
1211 08:57:42.351376 (+995301us) ts_tablet_manager.cc:1151] Deleting tablet ac6be204692a44fab2db551e79237299
1211 08:57:42.351412 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.351638 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":981406}
W20251211 08:57:42.357455 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet ac6be204692a44fab2db551e79237299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac6be204692a44fab2db551e79237299 already in progress: creating tablet
I20251211 08:57:42.357465 14604 raft_consensus.cc:2468] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
W20251211 08:57:42.358183 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 8ad600554c0946e0bcb0da6a42be2ffd because the tablet was not found. No further retry: Not found: Tablet not found: 8ad600554c0946e0bcb0da6a42be2ffd
W20251211 08:57:42.358757 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet db302282bc3d4d8cb56d1cbb2f73c4fe because the tablet was not found. No further retry: Not found: Tablet not found: db302282bc3d4d8cb56d1cbb2f73c4fe
I20251211 08:57:42.359033 15299 tablet_replica.cc:333] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:42.351753 15030 consensus_queue.cc:260] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:42.359297 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet a6570281e1064c79b6e2297723f28658 because the tablet was not found. No further retry: Not found: Tablet not found: a6570281e1064c79b6e2297723f28658
I20251211 08:57:42.359687 15299 raft_consensus.cc:2243] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251211 08:57:42.359792 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 42720f5f5bf445f2b72e513ad4a6d95f because the tablet was not found. No further retry: Not found: Tablet not found: 42720f5f5bf445f2b72e513ad4a6d95f
I20251211 08:57:42.360049 15299 raft_consensus.cc:2272] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:42.361466 15299 ts_tablet_manager.cc:1916] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.362512 15318 raft_consensus.cc:493] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:42.362877 15318 raft_consensus.cc:515] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.363051 14230 leader_election.cc:304] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:42.363994 15368 raft_consensus.cc:2804] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:42.364444 15318 leader_election.cc:290] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:42.364511 14904 ts_tablet_manager.cc:1950] T eb5a6e533e1f4da7b03a800d029f712d P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:42.365552 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 246) took 1314 ms. Trace:
I20251211 08:57:42.367609 14904 rpcz_store.cc:276] 1211 08:57:41.051622 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.051776 (+ 154us) service_pool.cc:225] Handling call
1211 08:57:42.276542 (+1224766us) ts_tablet_manager.cc:1151] Deleting tablet eb5a6e533e1f4da7b03a800d029f712d
1211 08:57:42.276585 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.311210 (+ 34625us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.350564 (+ 39354us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.365487 (+ 14923us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224292}
I20251211 08:57:42.368673 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet eb5a6e533e1f4da7b03a800d029f712d (table default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85]) successfully deleted
I20251211 08:57:42.369709 14904 tablet_replica.cc:333] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:42.370247 15030 ts_tablet_manager.cc:1434] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.053s user 0.001s sys 0.005s
I20251211 08:57:42.371309 14006 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_8216a7a3e6804a4f968cb39fc2c20793 [id=1b5eed1a6c2e4a99abcdbd9960bacb9a] alter complete (version 3)
I20251211 08:57:42.371681 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41d6ef975a47489385791065bb8f9f46" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:42.372639 14714 tablet_bootstrap.cc:654] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:42.370412 14904 raft_consensus.cc:2243] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:42.374665 14904 raft_consensus.cc:2272] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:42.375099 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 42720f5f5bf445f2b72e513ad4a6d95f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:42.376849 14006 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 33) took 781 ms (client timeout 1000 ms). Trace:
W20251211 08:57:42.378262 14544 leader_election.cc:343] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.342273 14905 raft_consensus.cc:2272] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:42.378978 14904 ts_tablet_manager.cc:1916] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.380667 14905 ts_tablet_manager.cc:1916] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.382575 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 42720f5f5bf445f2b72e513ad4a6d95f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:42.383124 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet db302282bc3d4d8cb56d1cbb2f73c4fe with delete_type TABLET_DATA_DELETED (replaced by tablet f785a3cc2e8145eebae7e9c2885b0b1e) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:42.382115 14006 rpcz_store.cc:269] 1211 08:57:41.595231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.596283 (+ 1052us) service_pool.cc:225] Handling call
1211 08:57:41.657237 (+ 60954us) spinlock_profiling.cc:243] Waited 46.5 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:42.376784 (+719547us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:42.241821 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:42.242140 (+ 319us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:42.242159 (+ 19us) write_op.cc:435] Acquired schema lock
1211 08:57:42.242169 (+ 10us) tablet.cc:660] Decoding operations
1211 08:57:42.259493 (+ 17324us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:42.259537 (+ 44us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:42.259561 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:42.259706 (+ 145us) tablet.cc:699] Row locks acquired
1211 08:57:42.259719 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:42.259828 (+ 109us) write_op.cc:270] Start()
1211 08:57:42.259914 (+ 86us) write_op.cc:276] Timestamp: P: 1765443462259919 usec, L: 0
1211 08:57:42.259928 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:42.260398 (+ 470us) log.cc:844] Serialized 356 byte log entry
1211 08:57:42.369428 (+109030us) op_driver.cc:464] REPLICATION: finished
1211 08:57:42.369681 (+ 253us) write_op.cc:301] APPLY: starting
1211 08:57:42.369738 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:42.369826 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:42.369909 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:42.370067 (+ 158us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:42.370181 (+ 114us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:42.370221 (+ 40us) write_op.cc:312] APPLY: finished
1211 08:57:42.370388 (+ 167us) log.cc:844] Serialized 23 byte log entry
1211 08:57:42.370970 (+ 582us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:42.371026 (+ 56us) write_op.cc:454] Released schema lock
1211 08:57:42.371056 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":14,"spinlock_wait_cycles":106923264,"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":1138,"apply.run_wall_time_us":5638,"num_ops":1,"prepare.queue_time_us":128787,"prepare.run_cpu_time_us":1761,"prepare.run_wall_time_us":27628,"replication_time_us":103090}]]}
I20251211 08:57:42.387272 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 2016208a33dc4a5cadd7ced8e85ccc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.389406 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b997f016aa90456c8c6b1a4dcf5f4c75" } modify_external_catalogs: true
I20251211 08:57:42.389847 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b997f016aa90456c8c6b1a4dcf5f4c75" } modify_external_catalogs: true
I20251211 08:57:42.390954 14722 tablet_bootstrap.cc:492] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:42.390990 14422 tablet_service.cc:1552] Processing DeleteTablet for tablet c53035e83710403bac3a2b401c8c97ad with delete_type TABLET_DATA_DELETED (replaced by tablet dd14de369e084828afb8a924c46c649b) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.391324 14722 ts_tablet_manager.cc:1403] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.301s user 0.010s sys 0.000s
I20251211 08:57:42.391556 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.392746 15368 raft_consensus.cc:697] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:42.392894 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_624d973765514fb29d96a32ce2dff370"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:42.393378 14722 raft_consensus.cc:359] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.393446 15368 consensus_queue.cc:237] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.393949 14722 raft_consensus.cc:385] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:42.394213 14722 raft_consensus.cc:740] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
W20251211 08:57:42.394237 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_624d973765514fb29d96a32ce2dff370 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:42.395408 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_74345a98ec224b2398937ca22336051b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:42.396765 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74345a98ec224b2398937ca22336051b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:42.401727 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e36c6e2868e143be8e7f72307202a268 consensus state has no leader
I20251211 08:57:42.402755 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b56b94bf721941859de71bcabd9b8b5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:42.403977 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b56b94bf721941859de71bcabd9b8b5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:42.404384 15122 raft_consensus.cc:493] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:42.404201 14722 consensus_queue.cc:260] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.404733 15122 raft_consensus.cc:515] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.406018 14722 ts_tablet_manager.cc:1434] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20251211 08:57:42.406401 15122 leader_election.cc:290] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:42.406919 14722 tablet_bootstrap.cc:492] T e8a8437b8b464fb8ab02a9c552f81db3 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:42.403846 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41d6ef975a47489385791065bb8f9f46" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:42.408234 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2016208a33dc4a5cadd7ced8e85ccc65" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:42.408208 14445 raft_consensus.cc:2468] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
W20251211 08:57:42.409565 14385 leader_election.cc:343] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.411825 15030 tablet_bootstrap.cc:492] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:42.413776 14542 leader_election.cc:304] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
W20251211 08:57:42.414789 14542 consensus_peers.cc:597] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 174fefd70bcc41418a77a5716f97cd5f. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:42.415632 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2016208a33dc4a5cadd7ced8e85ccc65" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:42.416195 14604 raft_consensus.cc:2468] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:42.417035 15318 raft_consensus.cc:2804] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:42.417419 15318 raft_consensus.cc:697] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:42.418157 15318 consensus_queue.cc:237] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.424149 14714 tablet_bootstrap.cc:492] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:42.424542 14714 ts_tablet_manager.cc:1403] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.157s user 0.007s sys 0.001s
I20251211 08:57:42.415272 14542 leader_election.cc:304] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:42.427266 14382 leader_election.cc:304] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:42.428108 15440 raft_consensus.cc:2804] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:42.428347 15122 raft_consensus.cc:2804] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:42.428725 15122 raft_consensus.cc:493] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:42.429032 15122 raft_consensus.cc:3060] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.430018 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_25fc063a532541f5a7d6c73d8f2f0e33" } new_table_name: "default.Table_9eac7c503ffa4e19bd39f541f2360217" modify_external_catalogs: true
I20251211 08:57:42.428436 15440 raft_consensus.cc:493] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:42.433326 15440 raft_consensus.cc:3060] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.433220 14714 raft_consensus.cc:359] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.433774 14714 raft_consensus.cc:385] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:42.437601 14722 tablet_bootstrap.cc:654] T e8a8437b8b464fb8ab02a9c552f81db3 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:42.446239 15122 raft_consensus.cc:515] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.448649 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2016208a33dc4a5cadd7ced8e85ccc65" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:42.449638 14385 leader_election.cc:343] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.454715 15122 leader_election.cc:290] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:42.454900 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2016208a33dc4a5cadd7ced8e85ccc65" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:42.455396 14604 raft_consensus.cc:3060] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.460302 15299 ts_tablet_manager.cc:1929] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:42.460615 15299 log.cc:1199] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/dca342fd9de241b5884600199f06fbeb
I20251211 08:57:42.461205 15299 ts_tablet_manager.cc:1950] T dca342fd9de241b5884600199f06fbeb P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:42.461711 14604 raft_consensus.cc:2468] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:42.467308 14382 leader_election.cc:304] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:42.473244 14714 raft_consensus.cc:740] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:42.474923 14904 ts_tablet_manager.cc:1929] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:42.475273 14904 log.cc:1199] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/81cfb3e0baef45ddab4a77a4e0790085
I20251211 08:57:42.475807 14904 ts_tablet_manager.cc:1950] T 81cfb3e0baef45ddab4a77a4e0790085 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:42.476452 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 249) took 1332 ms. Trace:
I20251211 08:57:42.476727 14904 rpcz_store.cc:276] 1211 08:57:41.144364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.144453 (+ 89us) service_pool.cc:225] Handling call
1211 08:57:42.369564 (+1225111us) ts_tablet_manager.cc:1151] Deleting tablet 81cfb3e0baef45ddab4a77a4e0790085
1211 08:57:42.369599 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.445017 (+ 75418us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.466689 (+ 21672us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.476407 (+ 9718us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224619}
I20251211 08:57:42.477906 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 81cfb3e0baef45ddab4a77a4e0790085 (table default.Table_02e79cde30df4a2d8e81657161c2a56e [id=8ac1575fb83e405493d250160ca6ac24]) successfully deleted
I20251211 08:57:42.478910 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 250) took 1311 ms. Trace:
I20251211 08:57:42.479046 15122 raft_consensus.cc:2804] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:42.476361 14714 consensus_queue.cc:260] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.479166 14904 rpcz_store.cc:276] 1211 08:57:41.167558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.167682 (+ 124us) service_pool.cc:225] Handling call
1211 08:57:42.478590 (+1310908us) ts_tablet_manager.cc:1151] Deleting tablet 080d1700df4a4df09acda65c6d021d96
1211 08:57:42.478625 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.478870 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1310389}
I20251211 08:57:42.479414 15122 raft_consensus.cc:697] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:42.480039 15122 consensus_queue.cc:237] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:42.481287 13999 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet ed0e90efe651499c929b1fb992712b96 (table default.Table_eb3b1d93a3ec4891a9d33b131661fd65 [id=46237116be8e46a68f51e20e008f374b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ed0e90efe651499c929b1fb992712b96 consensus state has no leader
I20251211 08:57:42.482164 15299 tablet_replica.cc:333] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:42.482664 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet dca342fd9de241b5884600199f06fbeb (table default.Table_f627382c623a4f818caf527706e776ab [id=22f3369559b34f5aa54c92c1bb01c2cb]) successfully deleted
I20251211 08:57:42.484241 15299 raft_consensus.cc:2243] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:42.485085 15299 raft_consensus.cc:2272] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:42.480098 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 080d1700df4a4df09acda65c6d021d96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 080d1700df4a4df09acda65c6d021d96 already in progress: creating tablet
I20251211 08:57:42.487336 15030 tablet_bootstrap.cc:654] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:42.487491 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 253) took 1140 ms. Trace:
I20251211 08:57:42.490954 15299 ts_tablet_manager.cc:1916] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.490298 14904 rpcz_store.cc:276] 1211 08:57:41.346916 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.347819 (+ 903us) service_pool.cc:225] Handling call
1211 08:57:42.487137 (+1139318us) ts_tablet_manager.cc:1151] Deleting tablet ac6be204692a44fab2db551e79237299
1211 08:57:42.487190 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.487434 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138194}
W20251211 08:57:42.492120 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet ac6be204692a44fab2db551e79237299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac6be204692a44fab2db551e79237299 already in progress: creating tablet
I20251211 08:57:42.492462 15443 leader_election.cc:290] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:42.494083 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 07aef1eea7c04d9cbf35c82e45df7a16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07aef1eea7c04d9cbf35c82e45df7a16 already in progress: creating tablet
I20251211 08:57:42.496151 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet 5daba165262e41af8d3481d3b4929092 with delete_type TABLET_DATA_DELETED (replaced by tablet da52cace2aab4f9dacef18ec5fa56aaf) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:42.499049 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ccfcacb5b294a628926d34cd31b137e with delete_type TABLET_DATA_DELETED (replaced by tablet 1065085c701f48c9ad7206c410ce5985) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.501277 14714 ts_tablet_manager.cc:1434] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.076s user 0.006s sys 0.001s
W20251211 08:57:42.501230 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a1b011cb7c647f79bb16ce1c92bcff5 already in progress: creating tablet
I20251211 08:57:42.505964 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eec7694087145d8a49dd93904f95980 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.507434 14714 tablet_bootstrap.cc:492] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:42.509053 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 570c7f08c46e4273bd38c32593dcf304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570c7f08c46e4273bd38c32593dcf304 already in progress: creating tablet
I20251211 08:57:42.514943 14904 tablet_replica.cc:333] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:42.515599 14904 raft_consensus.cc:2243] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:42.516021 14904 raft_consensus.cc:2272] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:42.527983 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet f909d82e0136470a810e1063d6769868 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:42.539089 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb2ff90b54544420b1aad9d2ab4132ff consensus state has no leader
I20251211 08:57:42.539202 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet 12379255548f49dfad17f3840a0c86bb from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:42.540659 14714 tablet_bootstrap.cc:654] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:42.538760 14905 ts_tablet_manager.cc:1929] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:42.548508 14905 log.cc:1199] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/4e410da32b3446edbf84b2eaa96424c3
I20251211 08:57:42.549218 14905 ts_tablet_manager.cc:1950] T 4e410da32b3446edbf84b2eaa96424c3 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:42.550555 14722 tablet_bootstrap.cc:492] T e8a8437b8b464fb8ab02a9c552f81db3 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:42.550949 14722 ts_tablet_manager.cc:1403] T e8a8437b8b464fb8ab02a9c552f81db3 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.144s user 0.007s sys 0.002s
I20251211 08:57:42.553009 14722 raft_consensus.cc:359] T e8a8437b8b464fb8ab02a9c552f81db3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.553587 14722 raft_consensus.cc:385] T e8a8437b8b464fb8ab02a9c552f81db3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:42.553791 14722 raft_consensus.cc:740] T e8a8437b8b464fb8ab02a9c552f81db3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:42.554358 14722 consensus_queue.cc:260] T e8a8437b8b464fb8ab02a9c552f81db3 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.563531 15299 ts_tablet_manager.cc:1929] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:42.563854 15299 log.cc:1199] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/e961c5a3b5f640cbaea78b6185048d04
I20251211 08:57:42.564400 15299 ts_tablet_manager.cc:1950] T e961c5a3b5f640cbaea78b6185048d04 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:42.566787 14012 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet e8a8437b8b464fb8ab02a9c552f81db3 (table default.Table_b19d002dfb154df6aadac72b8859cce9 [id=0944cc290d954cf4a33fec11acb53315]). Expected version 1 got 0
I20251211 08:57:42.567620 15440 raft_consensus.cc:515] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.570621 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41d6ef975a47489385791065bb8f9f46" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:42.571031 15443 raft_consensus.cc:493] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:42.571110 14445 raft_consensus.cc:3060] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.571425 15443 raft_consensus.cc:515] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.573474 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "578404732eef4e51b66b6aebffa87cdb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:42.573724 15443 leader_election.cc:290] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:42.574630 14544 leader_election.cc:343] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.577157 14445 raft_consensus.cc:2468] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:42.578716 14542 leader_election.cc:304] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:42.579512 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "578404732eef4e51b66b6aebffa87cdb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:42.580029 14448 raft_consensus.cc:2468] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:42.580360 15440 leader_election.cc:290] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:42.581122 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41d6ef975a47489385791065bb8f9f46" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:42.581974 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 296) took 1072 ms. Trace:
W20251211 08:57:42.582208 14544 leader_election.cc:343] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.582247 15299 rpcz_store.cc:276] 1211 08:57:41.509107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.509212 (+ 105us) service_pool.cc:225] Handling call
1211 08:57:42.481942 (+972730us) ts_tablet_manager.cc:1151] Deleting tablet e961c5a3b5f640cbaea78b6185048d04
1211 08:57:42.482012 (+ 70us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.530775 (+ 48763us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.562756 (+ 31981us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.581928 (+ 19172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":972155}
I20251211 08:57:42.583027 14542 leader_election.cc:304] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:42.583243 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet e961c5a3b5f640cbaea78b6185048d04 (table default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]) successfully deleted
I20251211 08:57:42.583834 15443 raft_consensus.cc:2804] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:42.584120 15443 raft_consensus.cc:493] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:42.584409 15443 raft_consensus.cc:3060] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.580840 15440 raft_consensus.cc:2804] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:42.587234 15440 raft_consensus.cc:697] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:42.588001 15440 consensus_queue.cc:237] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.589488 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 297) took 1039 ms. Trace:
I20251211 08:57:42.594616 15368 raft_consensus.cc:493] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:42.595043 15368 raft_consensus.cc:515] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.589762 15299 rpcz_store.cc:276] 1211 08:57:41.550129 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.550237 (+ 108us) service_pool.cc:225] Handling call
1211 08:57:42.589202 (+1038965us) ts_tablet_manager.cc:1151] Deleting tablet 07aef1eea7c04d9cbf35c82e45df7a16
1211 08:57:42.589236 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.589447 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038434}
I20251211 08:57:42.596271 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_5bb3e684c5f445188aa010ca4b32db87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:42.596736 15368 leader_election.cc:290] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:42.597748 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bb3e684c5f445188aa010ca4b32db87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:42.598995 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 07aef1eea7c04d9cbf35c82e45df7a16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07aef1eea7c04d9cbf35c82e45df7a16 already in progress: creating tablet
W20251211 08:57:42.599536 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a1b011cb7c647f79bb16ce1c92bcff5 already in progress: creating tablet
W20251211 08:57:42.600040 14230 leader_election.cc:343] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 5daba165262e41af8d3481d3b4929092
W20251211 08:57:42.600111 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 570c7f08c46e4273bd38c32593dcf304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570c7f08c46e4273bd38c32593dcf304 already in progress: creating tablet
I20251211 08:57:42.600783 15299 tablet_replica.cc:333] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:42.601728 15299 raft_consensus.cc:2243] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:42.601853 15443 raft_consensus.cc:515] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:42.602846 14231 leader_election.cc:343] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 5daba165262e41af8d3481d3b4929092
I20251211 08:57:42.603176 14231 leader_election.cc:304] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:42.603955 15368 raft_consensus.cc:2749] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:42.606101 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 247) took 1601 ms. Trace:
I20251211 08:57:42.606570 15299 raft_consensus.cc:2272] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:42.606374 14905 rpcz_store.cc:276] 1211 08:57:41.004560 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.004684 (+ 124us) service_pool.cc:225] Handling call
1211 08:57:42.339232 (+1334548us) ts_tablet_manager.cc:1151] Deleting tablet 4e410da32b3446edbf84b2eaa96424c3
1211 08:57:42.339280 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.430745 (+ 91465us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.517663 (+ 86918us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:42.606051 (+ 88388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333652}
I20251211 08:57:42.607077 15122 raft_consensus.cc:493] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:42.607502 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 4e410da32b3446edbf84b2eaa96424c3 (table default.Table_eb447b641c624aeba08c478e373a532d [id=38f5e9afc13a4797a7e8460f1f073280]) successfully deleted
I20251211 08:57:42.607401 15122 raft_consensus.cc:515] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.608306 14905 tablet_replica.cc:333] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:42.608343 15299 ts_tablet_manager.cc:1916] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.608949 15122 leader_election.cc:290] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:42.609011 14905 raft_consensus.cc:2243] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:42.609407 14905 raft_consensus.cc:2272] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:42.609769 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ccfcacb5b294a628926d34cd31b137e" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:42.610687 14385 leader_election.cc:343] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.610882 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ccfcacb5b294a628926d34cd31b137e" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:42.611883 14382 leader_election.cc:343] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.612207 14382 leader_election.cc:304] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:42.612955 15122 raft_consensus.cc:2749] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:42.612830 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f73c2258efaf4858800af6ac28f6d3a1" } new_table_name: "default.Table_da883b3cac244bc0aee81f7c17ac81a6" modify_external_catalogs: true
I20251211 08:57:42.616859 14904 ts_tablet_manager.cc:1916] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.618373 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 12379255548f49dfad17f3840a0c86bb
I20251211 08:57:42.619577 14013 catalog_manager.cc:6724] ReplaceTablet: tablet 12379255548f49dfad17f3840a0c86bb deleted and replaced by tablet f2dc6cb4efc744e6904728ad120df934
I20251211 08:57:42.620393 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 12379255548f49dfad17f3840a0c86bb with delete_type TABLET_DATA_DELETED (replaced by tablet f2dc6cb4efc744e6904728ad120df934) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.621016 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet 12379255548f49dfad17f3840a0c86bb with delete_type TABLET_DATA_DELETED (replaced by tablet f2dc6cb4efc744e6904728ad120df934) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:42.621925 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 12379255548f49dfad17f3840a0c86bb with delete_type TABLET_DATA_DELETED (replaced by tablet f2dc6cb4efc744e6904728ad120df934) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.630465 15030 tablet_bootstrap.cc:492] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:42.630968 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 145096302ca24d7e96f2fec45960b18a (DEFAULT_TABLE table=default.Table_eed8ebdeeca94232a29625f6435c6ea0 [id=68e99e0d80d245eeb62578a048e64f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.631979 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145096302ca24d7e96f2fec45960b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.636545 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 145096302ca24d7e96f2fec45960b18a (DEFAULT_TABLE table=default.Table_eed8ebdeeca94232a29625f6435c6ea0 [id=68e99e0d80d245eeb62578a048e64f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.637113 14008 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 41d6ef975a47489385791065bb8f9f46 (table default.Table_c9d62497a342488290749d4f7c02e748 [id=52ae4637827d424eb4d37362bf21cea2]). Expected version 2 got 0
I20251211 08:57:42.637404 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145096302ca24d7e96f2fec45960b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.639773 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 145096302ca24d7e96f2fec45960b18a (DEFAULT_TABLE table=default.Table_eed8ebdeeca94232a29625f6435c6ea0 [id=68e99e0d80d245eeb62578a048e64f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.640656 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145096302ca24d7e96f2fec45960b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.642432 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet 98d14ba15dcd47f8b9e8035b3c7d477c from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:42.639081 14008 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 578404732eef4e51b66b6aebffa87cdb (table default.Table_53b4b18528d343018f204b9a3b0f9e63 [id=26c0a9d9b1f84372b18a54f045eb768c]). Expected version 2 got 0
I20251211 08:57:42.651839 14722 ts_tablet_manager.cc:1434] T e8a8437b8b464fb8ab02a9c552f81db3 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.101s user 0.004s sys 0.000s
I20251211 08:57:42.652329 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 15595c5a432047de873c2647b78838d6 (DEFAULT_TABLE table=default.Table_cbfedb62a1204c96adec00226098c500 [id=874e5fb9e7d24220b6a41704988ca4a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.652662 14722 tablet_bootstrap.cc:492] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:42.653189 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15595c5a432047de873c2647b78838d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.655699 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 74c8dd2b036c42a0bec79d3637f00ef4 (DEFAULT_TABLE table=default.Table_2c8714fcc0964e3c8c96636192b9d6e1 [id=78961c2045bf45739d3632c0abf6325d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.655654 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 15595c5a432047de873c2647b78838d6 (DEFAULT_TABLE table=default.Table_cbfedb62a1204c96adec00226098c500 [id=874e5fb9e7d24220b6a41704988ca4a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.655824 14422 tablet_service.cc:1505] Processing CreateTablet for tablet 15595c5a432047de873c2647b78838d6 (DEFAULT_TABLE table=default.Table_cbfedb62a1204c96adec00226098c500 [id=874e5fb9e7d24220b6a41704988ca4a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.656502 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15595c5a432047de873c2647b78838d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.656672 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15595c5a432047de873c2647b78838d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.658591 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c8dd2b036c42a0bec79d3637f00ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.661005 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 74c8dd2b036c42a0bec79d3637f00ef4 (DEFAULT_TABLE table=default.Table_2c8714fcc0964e3c8c96636192b9d6e1 [id=78961c2045bf45739d3632c0abf6325d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.661804 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c8dd2b036c42a0bec79d3637f00ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.661702 14275 tablet_service.cc:1505] Processing CreateTablet for tablet c0bb339322504704ae57bba2f4c3a9bb (DEFAULT_TABLE table=default.Table_0e736d3e3e5f40dbbf25a5e615e5b842 [id=f48fec067bf941a4a62265c36fe526da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.662487 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0bb339322504704ae57bba2f4c3a9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.663416 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 74c8dd2b036c42a0bec79d3637f00ef4 (DEFAULT_TABLE table=default.Table_2c8714fcc0964e3c8c96636192b9d6e1 [id=78961c2045bf45739d3632c0abf6325d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.664294 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c8dd2b036c42a0bec79d3637f00ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.665761 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet dca342fd9de241b5884600199f06fbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.666617 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet c53035e83710403bac3a2b401c8c97ad with delete_type TABLET_DATA_DELETED (replaced by tablet dd14de369e084828afb8a924c46c649b) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.667388 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ad600554c0946e0bcb0da6a42be2ffd with delete_type TABLET_DATA_DELETED (replaced by tablet 6874fb018bc844b3aeeab7db8fc53e9a) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.667694 14420 tablet_service.cc:1505] Processing CreateTablet for tablet c0bb339322504704ae57bba2f4c3a9bb (DEFAULT_TABLE table=default.Table_0e736d3e3e5f40dbbf25a5e615e5b842 [id=f48fec067bf941a4a62265c36fe526da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.668594 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0bb339322504704ae57bba2f4c3a9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.670984 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:42.674157 13902 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 50627.7us
I20251211 08:57:42.683434 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "578404732eef4e51b66b6aebffa87cdb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:42.684837 14544 leader_election.cc:343] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:42.689318 14585 tablet_service.cc:1505] Processing CreateTablet for tablet c0bb339322504704ae57bba2f4c3a9bb (DEFAULT_TABLE table=default.Table_0e736d3e3e5f40dbbf25a5e615e5b842 [id=f48fec067bf941a4a62265c36fe526da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.690064 15443 leader_election.cc:290] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:42.690248 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0bb339322504704ae57bba2f4c3a9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.690203 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 4631dd0bc0c144e69a31de80de39a76a (DEFAULT_TABLE table=default.Table_f2a748aebc4d47738c1758ed749c0f98 [id=36535650771e4f23b249b046d913525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.691056 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4631dd0bc0c144e69a31de80de39a76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.691794 14905 ts_tablet_manager.cc:1916] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.696099 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "578404732eef4e51b66b6aebffa87cdb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:42.696703 14448 raft_consensus.cc:3060] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:42.702845 14268 tablet_service.cc:1505] Processing CreateTablet for tablet d69451d9e49740bb9af84d39d157c2f0 (DEFAULT_TABLE table=default.Table_299019253b214d92b585b6e128b24c3a [id=a70a9aab34f347a698c850cb21298eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.703814 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d69451d9e49740bb9af84d39d157c2f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.708158 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 4631dd0bc0c144e69a31de80de39a76a (DEFAULT_TABLE table=default.Table_f2a748aebc4d47738c1758ed749c0f98 [id=36535650771e4f23b249b046d913525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.709115 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4631dd0bc0c144e69a31de80de39a76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.709403 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet e8a8437b8b464fb8ab02a9c552f81db3 (table default.Table_b19d002dfb154df6aadac72b8859cce9 [id=0944cc290d954cf4a33fec11acb53315]). Expected version 1 got 0
I20251211 08:57:42.718534 14428 tablet_service.cc:1505] Processing CreateTablet for tablet d69451d9e49740bb9af84d39d157c2f0 (DEFAULT_TABLE table=default.Table_299019253b214d92b585b6e128b24c3a [id=a70a9aab34f347a698c850cb21298eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.719446 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d69451d9e49740bb9af84d39d157c2f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.635780 15030 ts_tablet_manager.cc:1403] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.224s user 0.009s sys 0.000s
I20251211 08:57:42.723707 15030 raft_consensus.cc:359] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.724267 15030 raft_consensus.cc:385] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:42.724478 15030 raft_consensus.cc:740] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:42.724985 15030 consensus_queue.cc:260] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.726370 15030 ts_tablet_manager.cc:1434] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251211 08:57:42.741019 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_0746d662358e4d9681b7d0a3529fb584"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:42.741992 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_c65937239ed54bb892938599e5f28043" } new_table_name: "default.Table_7158a7ed7bcb4153a059e26dbcd01a0e" modify_external_catalogs: true
W20251211 08:57:42.742374 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0746d662358e4d9681b7d0a3529fb584 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:42.742904 14448 raft_consensus.cc:2468] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:42.743119 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_840e4eee12064bf99a713fe71774ecc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:42.744134 14542 leader_election.cc:304] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
W20251211 08:57:42.744433 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_840e4eee12064bf99a713fe71774ecc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:42.745859 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_98031d7c370b4bfea07903c599528db5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:42.750571 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98031d7c370b4bfea07903c599528db5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:42.753132 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 4631dd0bc0c144e69a31de80de39a76a (DEFAULT_TABLE table=default.Table_f2a748aebc4d47738c1758ed749c0f98 [id=36535650771e4f23b249b046d913525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.754209 15443 raft_consensus.cc:2804] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:42.755179 15443 raft_consensus.cc:697] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:42.755877 15443 consensus_queue.cc:237] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.761526 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4631dd0bc0c144e69a31de80de39a76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.763124 14714 tablet_bootstrap.cc:492] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:42.767750 14714 ts_tablet_manager.cc:1403] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.266s user 0.008s sys 0.005s
I20251211 08:57:42.769948 14010 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 2ed430a2c82d498a9c3ca116eaac154e
I20251211 08:57:42.771368 15030 tablet_bootstrap.cc:492] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:42.772850 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet f56fd93aeac6435ea84388ce05580113
I20251211 08:57:42.774322 14904 ts_tablet_manager.cc:1929] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:42.775202 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 98d14ba15dcd47f8b9e8035b3c7d477c
I20251211 08:57:42.775661 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet f56fd93aeac6435ea84388ce05580113 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.775925 14904 log.cc:1199] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/78c6a703e1e44f41931fe465def01bf6
I20251211 08:57:42.772300 14575 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ed430a2c82d498a9c3ca116eaac154e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.776317 14013 catalog_manager.cc:6724] ReplaceTablet: tablet 98d14ba15dcd47f8b9e8035b3c7d477c deleted and replaced by tablet 8d39234b58734161b3bcfb67d0df1bd4
I20251211 08:57:42.776566 14904 ts_tablet_manager.cc:1950] T 78c6a703e1e44f41931fe465def01bf6 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:42.778422 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ed430a2c82d498a9c3ca116eaac154e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:42.781175 14575 tablet_service.cc:1552] Processing DeleteTablet for tablet 98d14ba15dcd47f8b9e8035b3c7d477c with delete_type TABLET_DATA_DELETED (replaced by tablet 8d39234b58734161b3bcfb67d0df1bd4) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.782877 14422 tablet_service.cc:1552] Processing DeleteTablet for tablet f56fd93aeac6435ea84388ce05580113 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.785471 14714 raft_consensus.cc:359] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.786088 14714 raft_consensus.cc:385] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:42.786315 14714 raft_consensus.cc:740] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:42.789031 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 98d14ba15dcd47f8b9e8035b3c7d477c with delete_type TABLET_DATA_DELETED (replaced by tablet 8d39234b58734161b3bcfb67d0df1bd4) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.790990 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 98d14ba15dcd47f8b9e8035b3c7d477c with delete_type TABLET_DATA_DELETED (replaced by tablet 8d39234b58734161b3bcfb67d0df1bd4) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:42.792428 14586 tablet_service.cc:1505] Processing CreateTablet for tablet d69451d9e49740bb9af84d39d157c2f0 (DEFAULT_TABLE table=default.Table_299019253b214d92b585b6e128b24c3a [id=a70a9aab34f347a698c850cb21298eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.793270 14722 tablet_bootstrap.cc:654] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:42.795393 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet f56fd93aeac6435ea84388ce05580113 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:42.797469 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d69451d9e49740bb9af84d39d157c2f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.799435 14428 tablet_service.cc:1505] Processing CreateTablet for tablet e90daf0f780a467e9c900b91c1615cb2 (DEFAULT_TABLE table=default.Table_07ea506691994cd3a2dc9606ed81de3f [id=afa43e27480743738f123b2b0660f51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.796128 14578 tablet_service.cc:1505] Processing CreateTablet for tablet e90daf0f780a467e9c900b91c1615cb2 (DEFAULT_TABLE table=default.Table_07ea506691994cd3a2dc9606ed81de3f [id=afa43e27480743738f123b2b0660f51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.800382 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90daf0f780a467e9c900b91c1615cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.800525 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90daf0f780a467e9c900b91c1615cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.802959 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ed430a2c82d498a9c3ca116eaac154e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.796801 14277 tablet_service.cc:1505] Processing CreateTablet for tablet e90daf0f780a467e9c900b91c1615cb2 (DEFAULT_TABLE table=default.Table_07ea506691994cd3a2dc9606ed81de3f [id=afa43e27480743738f123b2b0660f51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.816256 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90daf0f780a467e9c900b91c1615cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.818315 14008 catalog_manager.cc:5654] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:42.819572 14008 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 41d6ef975a47489385791065bb8f9f46 (table default.Table_c9d62497a342488290749d4f7c02e748 [id=52ae4637827d424eb4d37362bf21cea2]). Expected version 2 got 0
I20251211 08:57:42.822342 15030 tablet_bootstrap.cc:654] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:42.824185 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 2ed430a2c82d498a9c3ca116eaac154e (DEFAULT_TABLE table=default.Table_e444843f69064fbb9c2a6d0671fd9a2a [id=6c43f429406443d7b5834905929dcbd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.786875 14714 consensus_queue.cc:260] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.825347 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ed430a2c82d498a9c3ca116eaac154e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.827606 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 78c6a703e1e44f41931fe465def01bf6 (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]) successfully deleted
I20251211 08:57:42.831552 14904 tablet_replica.cc:333] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:42.840674 14904 raft_consensus.cc:2243] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:42.841274 14904 raft_consensus.cc:2272] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:42.851483 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_0e8a3c6f663b4834983142a353b03e40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:42.853056 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e8a3c6f663b4834983142a353b03e40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:42.854219 14722 tablet_bootstrap.cc:492] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:42.854524 14722 ts_tablet_manager.cc:1403] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.202s user 0.008s sys 0.000s
W20251211 08:57:42.855209 14544 consensus_peers.cc:597] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:42.858443 14904 ts_tablet_manager.cc:1916] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.859608 14722 raft_consensus.cc:359] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.860217 14722 raft_consensus.cc:385] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:42.860436 14722 raft_consensus.cc:740] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:42.872548 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b462cf21e3420c8e323a4ac4f04abe with delete_type TABLET_DATA_DELETED (replaced by tablet 2ed430a2c82d498a9c3ca116eaac154e) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:42.874351 14714 ts_tablet_manager.cc:1434] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.091s user 0.004s sys 0.000s
I20251211 08:57:42.880398 15299 ts_tablet_manager.cc:1929] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:42.880930 15299 log.cc:1199] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/78c6a703e1e44f41931fe465def01bf6
I20251211 08:57:42.881951 15299 ts_tablet_manager.cc:1950] T 78c6a703e1e44f41931fe465def01bf6 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:42.882579 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 2ed430a2c82d498a9c3ca116eaac154e (DEFAULT_TABLE table=default.Table_e444843f69064fbb9c2a6d0671fd9a2a [id=6c43f429406443d7b5834905929dcbd4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:42.883431 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 111adfd1ac1a42bdafa9416594e7e932 consensus state has no leader
I20251211 08:57:42.883541 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ed430a2c82d498a9c3ca116eaac154e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.883477 14277 tablet_service.cc:1505] Processing CreateTablet for tablet f6fb2096f80a43b5a5a6230a824fceb7 (DEFAULT_TABLE table=default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.884459 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6fb2096f80a43b5a5a6230a824fceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.884970 14427 tablet_service.cc:1505] Processing CreateTablet for tablet f6fb2096f80a43b5a5a6230a824fceb7 (DEFAULT_TABLE table=default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.885901 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6fb2096f80a43b5a5a6230a824fceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.891937 15030 tablet_bootstrap.cc:492] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:42.892311 15030 ts_tablet_manager.cc:1403] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.121s user 0.009s sys 0.000s
I20251211 08:57:42.894270 15030 raft_consensus.cc:359] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.897039 14722 consensus_queue.cc:260] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.900169 14722 ts_tablet_manager.cc:1434] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.045s user 0.006s sys 0.000s
I20251211 08:57:42.907868 15452 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251211 08:57:42.911376 14268 tablet_service.cc:1552] Processing DeleteTablet for tablet a6570281e1064c79b6e2297723f28658 with delete_type TABLET_DATA_DELETED (replaced by tablet e9905227c0a844d199362a442e7d630c) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:42.921640 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 2ed430a2c82d498a9c3ca116eaac154e (DEFAULT_TABLE table=default.Table_e444843f69064fbb9c2a6d0671fd9a2a [id=6c43f429406443d7b5834905929dcbd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:42.923506 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ed430a2c82d498a9c3ca116eaac154e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:42.924554 14008 catalog_manager.cc:5654] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:42.925745 14008 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 578404732eef4e51b66b6aebffa87cdb (table default.Table_53b4b18528d343018f204b9a3b0f9e63 [id=26c0a9d9b1f84372b18a54f045eb768c]). Expected version 2 got 0
I20251211 08:57:42.898886 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 2016208a33dc4a5cadd7ced8e85ccc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.932439 15030 raft_consensus.cc:385] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:42.932837 15030 raft_consensus.cc:740] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:42.933594 15030 consensus_queue.cc:260] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:42.945039 15299 tablet_replica.cc:333] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:42.946166 15299 raft_consensus.cc:2243] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:42.948881 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 78c6a703e1e44f41931fe465def01bf6 (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]) successfully deleted
I20251211 08:57:42.960830 15299 pending_rounds.cc:70] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:42.961097 15299 pending_rounds.cc:77] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256422340050944 op_type: NO_OP noop_request { }
I20251211 08:57:42.961416 15299 raft_consensus.cc:2889] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:42.961737 15299 raft_consensus.cc:2272] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:42.964998 14722 tablet_bootstrap.cc:492] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:42.968611 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ccfcacb5b294a628926d34cd31b137e with delete_type TABLET_DATA_DELETED (replaced by tablet 1065085c701f48c9ad7206c410ce5985) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:42.970966 15299 ts_tablet_manager.cc:1916] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:42.981182 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_da883b3cac244bc0aee81f7c17ac81a6" } modify_external_catalogs: true
I20251211 08:57:42.981614 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_da883b3cac244bc0aee81f7c17ac81a6" } modify_external_catalogs: true
I20251211 08:57:42.991359 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d9c4c3852105418a914f5055b9b70072" } modify_external_catalogs: true
I20251211 08:57:42.991837 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d9c4c3852105418a914f5055b9b70072" } modify_external_catalogs: true
I20251211 08:57:42.996057 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_3470f7a0edc3474884b6f95482ed49c9" } new_table_name: "default.Table_2bcb516a7ea04fe7be2fd23eb963dd27" modify_external_catalogs: true
I20251211 08:57:43.006498 14904 ts_tablet_manager.cc:1929] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:43.011089 14904 log.cc:1199] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/b8d753358f20476ba23d7d7b72e26e74
I20251211 08:57:43.011732 14904 ts_tablet_manager.cc:1950] T b8d753358f20476ba23d7d7b72e26e74 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:57:43.011653 14544 consensus_peers.cc:597] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:43.016906 14004 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da124c4afb5a402fb212d0ea95e0b47e has no consensus state
I20251211 08:57:43.018893 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_9bb3b4bd9d584bd6911f3b1cacc551df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:43.020474 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bb3b4bd9d584bd6911f3b1cacc551df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:43.024919 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 7af7c9ab27a64a1681f9874a922e986f
I20251211 08:57:43.027287 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 7af7c9ab27a64a1681f9874a922e986f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:43.027746 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 7af7c9ab27a64a1681f9874a922e986f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:43.028535 14296 raft_consensus.cc:3060] T e8a8437b8b464fb8ab02a9c552f81db3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:43.034646 14542 consensus_peers.cc:597] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dca342fd9de241b5884600199f06fbeb. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:43.035737 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 7af7c9ab27a64a1681f9874a922e986f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:43.041683 15030 ts_tablet_manager.cc:1434] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.149s user 0.014s sys 0.007s
W20251211 08:57:43.044929 14542 consensus_peers.cc:597] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e961c5a3b5f640cbaea78b6185048d04. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:43.046134 15122 raft_consensus.cc:493] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:43.046546 15122 raft_consensus.cc:515] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.048010 15030 tablet_bootstrap.cc:492] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:43.049950 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eec7694087145d8a49dd93904f95980" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:43.051347 14385 leader_election.cc:343] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:43.052469 15122 leader_election.cc:290] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:43.052656 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eec7694087145d8a49dd93904f95980" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:43.053210 14604 raft_consensus.cc:2468] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:43.053507 14714 tablet_bootstrap.cc:492] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:43.055382 15480 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251211 08:57:43.054071 14382 leader_election.cc:304] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:43.064280 15122 raft_consensus.cc:2804] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:43.064636 15122 raft_consensus.cc:493] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:43.064963 15122 raft_consensus.cc:3060] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:43.073822 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_8d772a8a54e2416dacc1df571fe6042c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:43.074083 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_064b00aac2ff440891fc0f5f0143a573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:43.075122 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d772a8a54e2416dacc1df571fe6042c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:43.075703 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_064b00aac2ff440891fc0f5f0143a573 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:43.087317 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7c905b0fe6de4e57bb2c49d7bf87206a" } new_table_name: "default.Table_d2367e8cf559401c8af423ce7425f5a8" modify_external_catalogs: true
I20251211 08:57:43.088253 14722 tablet_bootstrap.cc:654] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:43.095455 14904 tablet_replica.cc:333] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:43.096167 14904 raft_consensus.cc:2243] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:43.098434 14904 raft_consensus.cc:2272] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:43.100085 14904 ts_tablet_manager.cc:1916] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:43.100579 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet b8d753358f20476ba23d7d7b72e26e74 (table default.Table_6cbe43becdc846bcab4b1233f65cda0c [id=8a6e51e5adbf4ca39770244bfdc843bc]) successfully deleted
I20251211 08:57:43.101675 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_20ad075bb5ab4a5dadc32bcfe94b0322"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:43.103116 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20ad075bb5ab4a5dadc32bcfe94b0322 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:43.110453 15452 tablet.cc:1722] T e8a8437b8b464fb8ab02a9c552f81db3 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:43.110872 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 593df63696a04f85b00c301d93748e80
I20251211 08:57:43.112175 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 593df63696a04f85b00c301d93748e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:43.116195 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eec7694087145d8a49dd93904f95980 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:43.118119 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 593df63696a04f85b00c301d93748e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:43.118642 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 593df63696a04f85b00c301d93748e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:43.123821 15299 ts_tablet_manager.cc:1929] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:43.124155 15299 log.cc:1199] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/2016208a33dc4a5cadd7ced8e85ccc65
I20251211 08:57:43.124845 15299 ts_tablet_manager.cc:1950] T 2016208a33dc4a5cadd7ced8e85ccc65 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
W20251211 08:57:43.125636 14544 consensus_peers.cc:597] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:43.139286 14904 ts_tablet_manager.cc:1929] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:43.139621 14904 log.cc:1199] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/42720f5f5bf445f2b72e513ad4a6d95f
I20251211 08:57:43.140220 14904 ts_tablet_manager.cc:1950] T 42720f5f5bf445f2b72e513ad4a6d95f P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:43.141047 15299 tablet_replica.cc:333] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:43.141744 15299 raft_consensus.cc:2243] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:43.142180 15299 raft_consensus.cc:2272] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:43.143702 15299 ts_tablet_manager.cc:1916] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:43.147274 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 61775901e28946ddb723f4a01f90feb6 (DEFAULT_TABLE table=default.Table_50297110fe8f432b9202ae4060127653 [id=a571400822314bf9ac2d5213bdf68b3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.148243 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61775901e28946ddb723f4a01f90feb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.151245 14580 tablet_service.cc:1505] Processing CreateTablet for tablet f6fb2096f80a43b5a5a6230a824fceb7 (DEFAULT_TABLE table=default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.152287 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6fb2096f80a43b5a5a6230a824fceb7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:43.157150 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90c8960c5bfa4a08905bd211654e4798 consensus state has no leader
I20251211 08:57:43.117719 15030 tablet_bootstrap.cc:654] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:43.169855 14714 tablet_bootstrap.cc:654] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:43.169889 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 42720f5f5bf445f2b72e513ad4a6d95f (table default.Table_58593e18b9634b2c8912ff1109e05a52 [id=8bb6612dda374661a7d3a8a06bd230ae]) successfully deleted
I20251211 08:57:43.170619 14904 tablet_replica.cc:333] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:43.172451 14904 raft_consensus.cc:2243] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:43.173074 14904 pending_rounds.cc:70] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
I20251211 08:57:43.173275 14904 pending_rounds.cc:77] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256421971615744 op_type: NO_OP noop_request { }
I20251211 08:57:43.173548 14904 raft_consensus.cc:2889] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:43.173858 14904 raft_consensus.cc:2272] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:43.177109 14904 ts_tablet_manager.cc:1916] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:43.094340 15122 raft_consensus.cc:515] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.185003 14905 ts_tablet_manager.cc:1929] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:43.204511 15299 ts_tablet_manager.cc:1929] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:43.204891 15299 log.cc:1199] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/c53035e83710403bac3a2b401c8c97ad
I20251211 08:57:43.205477 15299 ts_tablet_manager.cc:1950] T c53035e83710403bac3a2b401c8c97ad P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:43.210446 14722 tablet_bootstrap.cc:492] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:43.210843 14722 ts_tablet_manager.cc:1403] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.246s user 0.012s sys 0.002s
I20251211 08:57:43.212787 14722 raft_consensus.cc:359] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.213335 14722 raft_consensus.cc:385] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:43.213570 15299 tablet_replica.cc:333] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:43.213610 14722 raft_consensus.cc:740] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:43.214282 15299 raft_consensus.cc:2243] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:43.220770 14011 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9eac7c503ffa4e19bd39f541f2360217" } new_table_name: "default.Table_31fcfbed4a2c4e8aa05fb42c261bf1d5" modify_external_catalogs: true
I20251211 08:57:43.214701 15299 raft_consensus.cc:2272] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:43.222623 15299 ts_tablet_manager.cc:1916] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:43.206110 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 61775901e28946ddb723f4a01f90feb6 (DEFAULT_TABLE table=default.Table_50297110fe8f432b9202ae4060127653 [id=a571400822314bf9ac2d5213bdf68b3a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:43.243286 14011 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb2ff90b54544420b1aad9d2ab4132ff consensus state has no leader
I20251211 08:57:43.243443 14268 tablet_service.cc:1505] Processing CreateTablet for tablet 2b96dc86ba944e5398d81975dfd7436d (DEFAULT_TABLE table=default.Table_b84ea7e1b576445e89c3d3fbd3ed631c [id=09f1f56235784e71b162bffd4f69e97f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.243749 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61775901e28946ddb723f4a01f90feb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.244300 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b96dc86ba944e5398d81975dfd7436d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.244994 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_840e4eee12064bf99a713fe71774ecc8" } modify_external_catalogs: true
I20251211 08:57:43.245445 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_840e4eee12064bf99a713fe71774ecc8" } modify_external_catalogs: true
I20251211 08:57:43.208395 15030 tablet_bootstrap.cc:492] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:43.251127 15030 ts_tablet_manager.cc:1403] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.205s user 0.008s sys 0.005s
I20251211 08:57:43.253432 15030 raft_consensus.cc:359] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.254155 15030 raft_consensus.cc:385] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:43.254438 15030 raft_consensus.cc:740] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:43.255139 15030 consensus_queue.cc:260] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.257583 15030 ts_tablet_manager.cc:1434] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251211 08:57:43.263391 15030 tablet_bootstrap.cc:492] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:43.263535 14004 master_service.cc:954] ReplaceTablet: received request to replace tablet dd14de369e084828afb8a924c46c649b from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:43.263835 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_9ec80b196de446bf8a12c0ae014f783e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:43.264674 14905 log.cc:1199] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/eb5a6e533e1f4da7b03a800d029f712d
W20251211 08:57:43.265197 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ec80b196de446bf8a12c0ae014f783e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:43.278973 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet dd14de369e084828afb8a924c46c649b
I20251211 08:57:43.279122 15122 leader_election.cc:290] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:43.280194 14004 catalog_manager.cc:6724] ReplaceTablet: tablet dd14de369e084828afb8a924c46c649b deleted and replaced by tablet ee50f29bfe674611a834d84a7c4e9070
I20251211 08:57:43.281906 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet dd14de369e084828afb8a924c46c649b with delete_type TABLET_DATA_DELETED (replaced by tablet ee50f29bfe674611a834d84a7c4e9070) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:43.282661 15299 ts_tablet_manager.cc:1929] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:43.283162 15299 log.cc:1199] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/9b5a4b6b1b28406c82e2acd9356b60d4
I20251211 08:57:43.284008 15299 ts_tablet_manager.cc:1950] T 9b5a4b6b1b28406c82e2acd9356b60d4 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:43.286510 15299 tablet_replica.cc:333] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:43.287086 14722 consensus_queue.cc:260] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.280966 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet dd14de369e084828afb8a924c46c649b with delete_type TABLET_DATA_DELETED (replaced by tablet ee50f29bfe674611a834d84a7c4e9070) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:43.293764 14905 ts_tablet_manager.cc:1950] T eb5a6e533e1f4da7b03a800d029f712d P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:43.295576 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 248) took 2246 ms. Trace:
I20251211 08:57:43.300969 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 2c0f8d7dd90f4beaae674fe827d37e0b (DEFAULT_TABLE table=default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.302037 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c0f8d7dd90f4beaae674fe827d37e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.311828 15030 tablet_bootstrap.cc:654] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:43.314476 14011 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 2 got 0
I20251211 08:57:43.316893 14007 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 61d4dfdecaf9402b9530424e6c0f30b0
I20251211 08:57:43.319087 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6b072b26d1564613a46376ccb5bb2635"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:43.320580 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b072b26d1564613a46376ccb5bb2635 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:43.322042 14011 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet e15483f539f24f1ba1b442663d66fd45 consensus state has no leader
I20251211 08:57:43.322232 14722 ts_tablet_manager.cc:1434] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.111s user 0.004s sys 0.001s
I20251211 08:57:43.324173 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_df3433d6d1fb4af39546e642816b6331" } modify_external_catalogs: true
I20251211 08:57:43.324677 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_df3433d6d1fb4af39546e642816b6331" } modify_external_catalogs: true
I20251211 08:57:43.330137 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 2c0f8d7dd90f4beaae674fe827d37e0b (DEFAULT_TABLE table=default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.331147 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c0f8d7dd90f4beaae674fe827d37e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.331156 15483 raft_consensus.cc:493] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:43.331533 15483 raft_consensus.cc:515] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.333177 15483 leader_election.cc:290] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:43.295857 14905 rpcz_store.cc:276] 1211 08:57:41.048983 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.049837 (+ 854us) service_pool.cc:225] Handling call
1211 08:57:42.608169 (+1558332us) ts_tablet_manager.cc:1151] Deleting tablet eb5a6e533e1f4da7b03a800d029f712d
1211 08:57:42.608213 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:42.808306 (+200093us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.184805 (+376499us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.295518 (+110713us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1557729}
I20251211 08:57:43.335909 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 251) took 2172 ms. Trace:
I20251211 08:57:43.336164 14905 rpcz_store.cc:276] 1211 08:57:41.163665 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.163804 (+ 139us) service_pool.cc:225] Handling call
1211 08:57:43.335584 (+2171780us) ts_tablet_manager.cc:1151] Deleting tablet 080d1700df4a4df09acda65c6d021d96
1211 08:57:43.335628 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.335865 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2171240}
I20251211 08:57:43.337124 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 255) took 2022 ms. Trace:
I20251211 08:57:43.337424 14905 rpcz_store.cc:276] 1211 08:57:41.314913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.315392 (+ 479us) service_pool.cc:225] Handling call
1211 08:57:43.336838 (+2021446us) ts_tablet_manager.cc:1151] Deleting tablet 9168007ca0584954a734e2c8f2e73f70
1211 08:57:43.336917 (+ 79us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.337083 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4864,"tablet-delete.queue_time_us":2020910}
I20251211 08:57:43.340622 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eec7694087145d8a49dd93904f95980" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:43.341037 14607 raft_consensus.cc:3060] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:43.343070 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5b462cf21e3420c8e323a4ac4f04abe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:43.344286 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet eb5a6e533e1f4da7b03a800d029f712d (table default.Table_91bdd3847f834c4b8500113d994514a5 [id=1e865cf903a347eaa103484c3a486d85]) successfully deleted
W20251211 08:57:43.344892 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 080d1700df4a4df09acda65c6d021d96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 080d1700df4a4df09acda65c6d021d96 already in progress: creating tablet
W20251211 08:57:43.345358 14382 leader_election.cc:343] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:43.354899 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 242505us
I20251211 08:57:43.355397 15299 raft_consensus.cc:2243] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:43.355517 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 2016208a33dc4a5cadd7ced8e85ccc65 (table default.Table_face70f737874a76a60d1f4ea105380c [id=f93e9bee7bdd443c9eb3d158f1e40d06]) successfully deleted
I20251211 08:57:43.355863 15299 raft_consensus.cc:2272] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:43.355738 14607 raft_consensus.cc:2468] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:43.356276 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet c53035e83710403bac3a2b401c8c97ad (table default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]) successfully deleted
I20251211 08:57:43.357604 15299 ts_tablet_manager.cc:1916] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:43.358912 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eec7694087145d8a49dd93904f95980" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:43.357470 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_5df60df88ddd40deb8310ab9f413aab5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:43.362423 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 9b5a4b6b1b28406c82e2acd9356b60d4 (table default.Table_baaa95f73b8742388a79269d898a5a17 [id=99243463ed7545ff80f928e6664a3748]) successfully deleted
I20251211 08:57:43.363654 14299 raft_consensus.cc:3060] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:43.364915 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5b462cf21e3420c8e323a4ac4f04abe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:43.367799 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5df60df88ddd40deb8310ab9f413aab5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:43.371400 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_2c8714fcc0964e3c8c96636192b9d6e1" } new_table_name: "default.Table_1a66b3d9fba24b2e9415fb8743297c0f" modify_external_catalogs: true
W20251211 08:57:43.374485 14542 consensus_peers.cc:597] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b5a4b6b1b28406c82e2acd9356b60d4. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:43.377557 14585 tablet_service.cc:1505] Processing CreateTablet for tablet a2530df4933249e19d96759910a47673 (DEFAULT_TABLE table=default.Table_84155c7aff0d471687b30b86e434adf6 [id=4b2043efe1174e199516c89f4d5e16df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.378474 14268 tablet_service.cc:1505] Processing CreateTablet for tablet 1bec0d4eaf4f4569ba4bfc78c2decb4d (DEFAULT_TABLE table=default.Table_e1af8fbbea91449781ac2466fe85e5c0 [id=515bd06e1ad340e1a331ec993e30b32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.379424 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bec0d4eaf4f4569ba4bfc78c2decb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.383167 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f819a255bf2a4b3ebcddaa81435bc4e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:43.384508 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f819a255bf2a4b3ebcddaa81435bc4e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:43.386900 15030 tablet_bootstrap.cc:492] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:43.387401 15030 ts_tablet_manager.cc:1403] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.124s user 0.008s sys 0.001s
I20251211 08:57:43.389855 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 2b96dc86ba944e5398d81975dfd7436d (DEFAULT_TABLE table=default.Table_b84ea7e1b576445e89c3d3fbd3ed631c [id=09f1f56235784e71b162bffd4f69e97f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.390801 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b96dc86ba944e5398d81975dfd7436d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.390801 14382 leader_election.cc:304] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:43.393155 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2530df4933249e19d96759910a47673. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.393635 15122 raft_consensus.cc:2804] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:43.394640 15443 raft_consensus.cc:493] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:43.394626 15122 raft_consensus.cc:697] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:43.395056 15443 raft_consensus.cc:515] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.395300 15122 consensus_queue.cc:237] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.396814 15443 leader_election.cc:290] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:43.398665 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ccfcacb5b294a628926d34cd31b137e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:43.399935 14544 leader_election.cc:343] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:43.401155 14277 tablet_service.cc:1505] Processing CreateTablet for tablet a2530df4933249e19d96759910a47673 (DEFAULT_TABLE table=default.Table_84155c7aff0d471687b30b86e434adf6 [id=4b2043efe1174e199516c89f4d5e16df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.402693 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 2c0f8d7dd90f4beaae674fe827d37e0b (DEFAULT_TABLE table=default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:43.403429 14385 leader_election.cc:343] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:43.403607 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c0f8d7dd90f4beaae674fe827d37e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.403419 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 2b96dc86ba944e5398d81975dfd7436d (DEFAULT_TABLE table=default.Table_b84ea7e1b576445e89c3d3fbd3ed631c [id=09f1f56235784e71b162bffd4f69e97f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.404310 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b96dc86ba944e5398d81975dfd7436d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.407055 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 256) took 2091 ms. Trace:
I20251211 08:57:43.407096 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ccfcacb5b294a628926d34cd31b137e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:43.407011 14422 tablet_service.cc:1505] Processing CreateTablet for tablet a2530df4933249e19d96759910a47673 (DEFAULT_TABLE table=default.Table_84155c7aff0d471687b30b86e434adf6 [id=4b2043efe1174e199516c89f4d5e16df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.407382 14905 rpcz_store.cc:276] 1211 08:57:41.315225 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.316021 (+ 796us) service_pool.cc:225] Handling call
1211 08:57:43.383037 (+2067016us) ts_tablet_manager.cc:1151] Deleting tablet 81cfb3e0baef45ddab4a77a4e0790085
1211 08:57:43.406839 (+ 23802us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.407008 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2066503}
I20251211 08:57:43.408015 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2530df4933249e19d96759910a47673. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.408422 14905 tablet_replica.cc:333] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:43.408377 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 61775901e28946ddb723f4a01f90feb6 (DEFAULT_TABLE table=default.Table_50297110fe8f432b9202ae4060127653 [id=a571400822314bf9ac2d5213bdf68b3a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:43.408589 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 81cfb3e0baef45ddab4a77a4e0790085 because the tablet was not found. No further retry: Not found: Tablet not found: 81cfb3e0baef45ddab4a77a4e0790085
I20251211 08:57:43.409070 14905 raft_consensus.cc:2243] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:43.409180 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61775901e28946ddb723f4a01f90feb6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:43.409210 14542 leader_election.cc:343] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: Tablet not RUNNING: STOPPED
I20251211 08:57:43.409458 14905 raft_consensus.cc:2272] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:43.409507 14542 leader_election.cc:304] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:43.410210 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 1bec0d4eaf4f4569ba4bfc78c2decb4d (DEFAULT_TABLE table=default.Table_e1af8fbbea91449781ac2466fe85e5c0 [id=515bd06e1ad340e1a331ec993e30b32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.410322 15443 raft_consensus.cc:2749] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:43.411033 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bec0d4eaf4f4569ba4bfc78c2decb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.413707 15493 tablet.cc:1722] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251211 08:57:43.411238 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2530df4933249e19d96759910a47673. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.411818 14722 tablet_bootstrap.cc:492] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:43.411021 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet dd14de369e084828afb8a924c46c649b with delete_type TABLET_DATA_DELETED (replaced by tablet ee50f29bfe674611a834d84a7c4e9070) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:43.429911 14714 tablet_bootstrap.cc:492] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:43.430261 14714 ts_tablet_manager.cc:1403] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.377s user 0.006s sys 0.006s
W20251211 08:57:43.430474 14385 leader_election.cc:343] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:43.430821 14385 leader_election.cc:304] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:43.431567 14904 ts_tablet_manager.cc:1929] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:43.431476 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 97b12636e7aa4decb02e275a5ca28dbf (DEFAULT_TABLE table=default.Table_010b1e9526c341ef81e1805cb6929e30 [id=ce5ba2a646fe4fb3ac5ed9911d1e5a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.431880 14904 log.cc:1199] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/db302282bc3d4d8cb56d1cbb2f73c4fe
I20251211 08:57:43.432536 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b12636e7aa4decb02e275a5ca28dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.441295 14722 tablet_bootstrap.cc:654] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:43.446426 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_3ec948949bdb4957b702fc9f5b18207a" } new_table_name: "default.Table_752a7e4e23044981af7db22be260121e" modify_external_catalogs: true
I20251211 08:57:43.452190 15030 raft_consensus.cc:359] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.452819 15030 raft_consensus.cc:385] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:43.453049 15030 raft_consensus.cc:740] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:43.453603 15030 consensus_queue.cc:260] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:43.457381 14544 consensus_peers.cc:597] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:43.461227 14905 ts_tablet_manager.cc:1916] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:43.462734 15484 raft_consensus.cc:2749] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:43.464421 14005 master_service.cc:954] ReplaceTablet: received request to replace tablet 74c8dd2b036c42a0bec79d3637f00ef4 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:43.465570 15484 raft_consensus.cc:493] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:43.465965 15484 raft_consensus.cc:515] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.467787 15484 leader_election.cc:290] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:43.468456 14714 raft_consensus.cc:359] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.468968 14714 raft_consensus.cc:385] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:43.469204 14714 raft_consensus.cc:740] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:43.469683 14714 consensus_queue.cc:260] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.471845 14714 ts_tablet_manager.cc:1434] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20251211 08:57:43.472756 14714 tablet_bootstrap.cc:492] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:43.485029 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 2 got 0
W20251211 08:57:43.489567 13997 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet e15483f539f24f1ba1b442663d66fd45 consensus state has no leader
I20251211 08:57:43.495158 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 1bec0d4eaf4f4569ba4bfc78c2decb4d (DEFAULT_TABLE table=default.Table_e1af8fbbea91449781ac2466fe85e5c0 [id=515bd06e1ad340e1a331ec993e30b32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.496061 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 97b12636e7aa4decb02e275a5ca28dbf (DEFAULT_TABLE table=default.Table_010b1e9526c341ef81e1805cb6929e30 [id=ce5ba2a646fe4fb3ac5ed9911d1e5a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.497236 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bec0d4eaf4f4569ba4bfc78c2decb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.499136 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f909d82e0136470a810e1063d6769868" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:43.499642 14607 raft_consensus.cc:2468] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:43.500669 14382 leader_election.cc:304] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:43.501060 14904 ts_tablet_manager.cc:1950] T db302282bc3d4d8cb56d1cbb2f73c4fe P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:43.504678 15030 ts_tablet_manager.cc:1434] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.117s user 0.006s sys 0.001s
I20251211 08:57:43.504827 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 578404732eef4e51b66b6aebffa87cdb (table default.Table_53b4b18528d343018f204b9a3b0f9e63 [id=26c0a9d9b1f84372b18a54f045eb768c]). Expected version 2 got 0
I20251211 08:57:43.509954 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f909d82e0136470a810e1063d6769868" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:43.510426 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b12636e7aa4decb02e275a5ca28dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.511056 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_eb05ea73b5a7414e9914f98266240d00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:43.512396 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb05ea73b5a7414e9914f98266240d00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:43.515853 14542 consensus_peers.cc:597] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c53035e83710403bac3a2b401c8c97ad. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:43.516330 15030 tablet_bootstrap.cc:492] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:43.516598 15484 raft_consensus.cc:2804] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:43.517763 15484 raft_consensus.cc:493] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:43.518069 15484 raft_consensus.cc:3060] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:43.519585 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_0ff1ebe3d6b144cc94137467d38b30bb" } modify_external_catalogs: true
I20251211 08:57:43.522851 14445 raft_consensus.cc:1275] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Refusing update from remote peer c5e624bcfd154cd586ef1dedb9940063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251211 08:57:43.523301 14385 leader_election.cc:343] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:43.524113 14722 tablet_bootstrap.cc:492] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:43.524302 15443 consensus_queue.cc:1048] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:43.524534 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 97b12636e7aa4decb02e275a5ca28dbf (DEFAULT_TABLE table=default.Table_010b1e9526c341ef81e1805cb6929e30 [id=ce5ba2a646fe4fb3ac5ed9911d1e5a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:43.525803 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b12636e7aa4decb02e275a5ca28dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:43.524788 14722 ts_tablet_manager.cc:1403] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.113s user 0.008s sys 0.009s
I20251211 08:57:43.528601 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 41d6ef975a47489385791065bb8f9f46 (table default.Table_c9d62497a342488290749d4f7c02e748 [id=52ae4637827d424eb4d37362bf21cea2]). Expected version 2 got 0
I20251211 08:57:43.540809 14722 raft_consensus.cc:359] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.541245 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 273) took 1159 ms. Trace:
I20251211 08:57:43.541656 14904 rpcz_store.cc:276] 1211 08:57:42.382321 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.383014 (+ 693us) service_pool.cc:225] Handling call
1211 08:57:43.170484 (+787470us) ts_tablet_manager.cc:1151] Deleting tablet db302282bc3d4d8cb56d1cbb2f73c4fe
1211 08:57:43.170521 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.238027 (+ 67506us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.431392 (+193365us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.541196 (+109804us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":787026}
I20251211 08:57:43.543619 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet db302282bc3d4d8cb56d1cbb2f73c4fe (table default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]) successfully deleted
I20251211 08:57:43.547040 14904 tablet_replica.cc:333] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:43.547945 14904 raft_consensus.cc:2243] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:43.548372 14904 raft_consensus.cc:2272] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:43.549738 14904 ts_tablet_manager.cc:1916] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:43.550279 15299 ts_tablet_manager.cc:1929] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:43.550606 15299 log.cc:1199] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/9ccfcacb5b294a628926d34cd31b137e
I20251211 08:57:43.567059 15299 ts_tablet_manager.cc:1950] T 9ccfcacb5b294a628926d34cd31b137e P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:43.583250 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_0ff1ebe3d6b144cc94137467d38b30bb" } modify_external_catalogs: true
I20251211 08:57:43.587145 14007 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet b3b7fae7d7554f1d9d48cd1b4dc2fc10
W20251211 08:57:43.595923 14544 consensus_peers.cc:597] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:43.606077 15443 raft_consensus.cc:493] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:43.606557 15443 raft_consensus.cc:515] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.608390 15443 leader_election.cc:290] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:43.609282 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 314) took 1110 ms. Trace:
I20251211 08:57:43.609563 15299 rpcz_store.cc:276] 1211 08:57:42.498815 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.498930 (+ 115us) service_pool.cc:225] Handling call
1211 08:57:43.286337 (+787407us) ts_tablet_manager.cc:1151] Deleting tablet 9ccfcacb5b294a628926d34cd31b137e
1211 08:57:43.286384 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.496932 (+210548us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.550069 (+ 53137us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.609221 (+ 59152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40383,"tablet-delete.queue_time_us":786943}
I20251211 08:57:43.610581 15299 tablet_replica.cc:333] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:43.610603 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e15483f539f24f1ba1b442663d66fd45" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:43.610905 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 9ccfcacb5b294a628926d34cd31b137e (table default.Table_93954115dc5d4d2283faa424e83608b0 [id=cb30302a7db04098ae9a5a92ce055d40]) successfully deleted
W20251211 08:57:43.611747 14544 leader_election.cc:343] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:43.620983 15030 tablet_bootstrap.cc:654] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:43.633760 15484 raft_consensus.cc:515] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.636003 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e15483f539f24f1ba1b442663d66fd45" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:43.638571 15030 tablet_bootstrap.cc:492] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:43.638965 15030 ts_tablet_manager.cc:1403] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.123s user 0.007s sys 0.004s
I20251211 08:57:43.541391 14722 raft_consensus.cc:385] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:43.641070 14722 raft_consensus.cc:740] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
W20251211 08:57:43.641402 14542 leader_election.cc:343] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:43.641760 14542 leader_election.cc:304] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:43.641733 14722 consensus_queue.cc:260] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.642371 15443 raft_consensus.cc:2749] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:43.642833 15443 consensus_queue.cc:1048] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:43.646049 14714 tablet_bootstrap.cc:654] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:43.648938 15030 raft_consensus.cc:359] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.649523 15030 raft_consensus.cc:385] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:43.649760 15030 raft_consensus.cc:740] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:43.651700 15030 consensus_queue.cc:260] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.657687 15030 ts_tablet_manager.cc:1434] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20251211 08:57:43.660349 14722 ts_tablet_manager.cc:1434] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.133s user 0.005s sys 0.000s
I20251211 08:57:43.662645 14722 tablet_bootstrap.cc:492] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:43.665891 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f909d82e0136470a810e1063d6769868" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:43.666944 14385 leader_election.cc:343] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:43.667757 15484 leader_election.cc:290] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:43.669879 15030 tablet_bootstrap.cc:492] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
W20251211 08:57:43.676379 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74c8dd2b036c42a0bec79d3637f00ef4 consensus state has no leader
I20251211 08:57:43.687978 14905 ts_tablet_manager.cc:1929] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:43.701218 14904 ts_tablet_manager.cc:1929] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:43.701592 14904 log.cc:1199] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/5daba165262e41af8d3481d3b4929092
I20251211 08:57:43.702175 14904 ts_tablet_manager.cc:1950] T 5daba165262e41af8d3481d3b4929092 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:43.703243 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet f909d82e0136470a810e1063d6769868 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:43.712605 14268 tablet_service.cc:1552] Processing DeleteTablet for tablet 174fefd70bcc41418a77a5716f97cd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:43.722263 14544 consensus_peers.cc:597] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:43.723693 14722 tablet_bootstrap.cc:654] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:43.688386 14905 log.cc:1199] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/db302282bc3d4d8cb56d1cbb2f73c4fe
I20251211 08:57:43.724831 14905 ts_tablet_manager.cc:1950] T db302282bc3d4d8cb56d1cbb2f73c4fe P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:43.726576 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 258) took 2390 ms. Trace:
I20251211 08:57:43.726851 14905 rpcz_store.cc:276] 1211 08:57:41.335798 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.335920 (+ 122us) service_pool.cc:225] Handling call
1211 08:57:43.408261 (+2072341us) ts_tablet_manager.cc:1151] Deleting tablet db302282bc3d4d8cb56d1cbb2f73c4fe
1211 08:57:43.408310 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.625228 (+216918us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.687765 (+ 62537us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.726524 (+ 38759us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2071755}
I20251211 08:57:43.728212 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet db302282bc3d4d8cb56d1cbb2f73c4fe (table default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]) successfully deleted
I20251211 08:57:43.728927 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 5f7775994f684326b6312c802f6b084d
I20251211 08:57:43.731303 14268 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7775994f684326b6312c802f6b084d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:43.733438 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7775994f684326b6312c802f6b084d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:43.740257 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7775994f684326b6312c802f6b084d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:43.741505 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 259) took 2398 ms. Trace:
I20251211 08:57:43.741935 14905 rpcz_store.cc:276] 1211 08:57:41.342624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.342744 (+ 120us) service_pool.cc:225] Handling call
1211 08:57:43.741072 (+2398328us) ts_tablet_manager.cc:1151] Deleting tablet ac6be204692a44fab2db551e79237299
1211 08:57:43.741130 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.741446 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2397783}
W20251211 08:57:43.744571 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet ac6be204692a44fab2db551e79237299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac6be204692a44fab2db551e79237299 already in progress: creating tablet
I20251211 08:57:43.745775 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 264) took 2197 ms. Trace:
I20251211 08:57:43.746369 14905 rpcz_store.cc:276] 1211 08:57:41.548395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:41.548549 (+ 154us) service_pool.cc:225] Handling call
1211 08:57:43.745445 (+2196896us) ts_tablet_manager.cc:1151] Deleting tablet 07aef1eea7c04d9cbf35c82e45df7a16
1211 08:57:43.745501 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.745732 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2196267}
W20251211 08:57:43.747536 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 07aef1eea7c04d9cbf35c82e45df7a16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07aef1eea7c04d9cbf35c82e45df7a16 already in progress: creating tablet
I20251211 08:57:43.748432 14905 tablet_replica.cc:333] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:57:43.751202 14013 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 143f20f5c37948068ff2fb0d92f0f34b consensus state has no leader
I20251211 08:57:43.751781 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 274) took 1207 ms. Trace:
I20251211 08:57:43.752102 14904 rpcz_store.cc:276] 1211 08:57:42.495344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.496018 (+ 674us) service_pool.cc:225] Handling call
1211 08:57:43.543790 (+1047772us) ts_tablet_manager.cc:1151] Deleting tablet 5daba165262e41af8d3481d3b4929092
1211 08:57:43.546885 (+ 3095us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.614629 (+ 67744us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.701008 (+ 86379us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.702804 (+ 1796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047276}
I20251211 08:57:43.752832 15299 raft_consensus.cc:2243] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:43.754134 15299 pending_rounds.cc:70] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:43.754328 15299 pending_rounds.cc:77] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256426473164800 op_type: NO_OP noop_request { }
I20251211 08:57:43.754624 15299 raft_consensus.cc:2889] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:43.754909 15299 raft_consensus.cc:2272] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:43.758196 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 5daba165262e41af8d3481d3b4929092 (table default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]) successfully deleted
W20251211 08:57:43.765910 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 156206us
I20251211 08:57:43.769449 14714 tablet_bootstrap.cc:492] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:43.759061 14904 tablet_replica.cc:333] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:43.769821 14714 ts_tablet_manager.cc:1403] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.297s user 0.002s sys 0.011s
I20251211 08:57:43.770815 14904 raft_consensus.cc:2243] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:43.771679 14904 raft_consensus.cc:2272] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:43.776180 15299 ts_tablet_manager.cc:1916] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:43.783144 14904 ts_tablet_manager.cc:1916] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:43.767104 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f909d82e0136470a810e1063d6769868" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:43.783447 15030 tablet_bootstrap.cc:654] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:43.783792 14607 raft_consensus.cc:3060] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:43.784170 14905 raft_consensus.cc:2243] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:43.784787 14905 raft_consensus.cc:2272] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:43.786844 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 74c8dd2b036c42a0bec79d3637f00ef4
I20251211 08:57:43.787158 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_d2111e461ff04909b0f4f135f0470016"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:43.787757 14905 ts_tablet_manager.cc:1916] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:43.788033 14005 catalog_manager.cc:6724] ReplaceTablet: tablet 74c8dd2b036c42a0bec79d3637f00ef4 deleted and replaced by tablet 8c1e2d6ba1014d00a6df8c6353cff3ab
W20251211 08:57:43.789263 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2111e461ff04909b0f4f135f0470016 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:43.789975 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 74c8dd2b036c42a0bec79d3637f00ef4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c1e2d6ba1014d00a6df8c6353cff3ab) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:43.790136 14607 raft_consensus.cc:2468] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:43.791039 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 74c8dd2b036c42a0bec79d3637f00ef4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c1e2d6ba1014d00a6df8c6353cff3ab) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:43.791193 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_cfa8f9f640e54badacad03895e72ca57" } modify_external_catalogs: true
I20251211 08:57:43.791652 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_cfa8f9f640e54badacad03895e72ca57" } modify_external_catalogs: true
I20251211 08:57:43.806784 14007 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 41d6ef975a47489385791065bb8f9f46 (table default.Table_c9d62497a342488290749d4f7c02e748 [id=52ae4637827d424eb4d37362bf21cea2]). Expected version 2 got 0
I20251211 08:57:43.810395 15299 ts_tablet_manager.cc:1929] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:43.813666 15299 log.cc:1199] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/4eec7694087145d8a49dd93904f95980
I20251211 08:57:43.814263 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet c112dfcc301648cda6f0f038b1f2af83
I20251211 08:57:43.810648 14382 leader_election.cc:304] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:43.815191 15506 raft_consensus.cc:2804] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:43.816295 15506 raft_consensus.cc:697] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:43.817086 15506 consensus_queue.cc:237] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.820694 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet c112dfcc301648cda6f0f038b1f2af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:43.823689 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet c112dfcc301648cda6f0f038b1f2af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:43.823684 14714 raft_consensus.cc:359] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.825515 14714 raft_consensus.cc:385] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:43.825507 14005 master_service.cc:954] ReplaceTablet: received request to replace tablet 07f983604bfa474f9c9ce0c982d738fc from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:43.825755 14714 raft_consensus.cc:740] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:43.826148 15299 ts_tablet_manager.cc:1950] T 4eec7694087145d8a49dd93904f95980 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:43.829661 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 315) took 1324 ms. Trace:
I20251211 08:57:43.826265 14714 consensus_queue.cc:260] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.780150 14722 tablet_bootstrap.cc:492] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:43.790988 14268 tablet_service.cc:1552] Processing DeleteTablet for tablet 74c8dd2b036c42a0bec79d3637f00ef4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c1e2d6ba1014d00a6df8c6353cff3ab) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:43.812980 15368 raft_consensus.cc:493] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:43.835256 15368 raft_consensus.cc:515] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.837051 15368 leader_election.cc:290] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:43.815653 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet c112dfcc301648cda6f0f038b1f2af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:43.838125 14714 ts_tablet_manager.cc:1434] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.068s user 0.006s sys 0.000s
I20251211 08:57:43.838727 14722 ts_tablet_manager.cc:1403] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.178s user 0.009s sys 0.000s
I20251211 08:57:43.840158 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6570281e1064c79b6e2297723f28658" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:43.840785 14607 raft_consensus.cc:2468] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:43.841012 14722 raft_consensus.cc:359] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.841573 14722 raft_consensus.cc:385] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:43.841809 14722 raft_consensus.cc:740] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:43.841713 14230 leader_election.cc:304] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:43.842515 15368 raft_consensus.cc:2804] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:43.842834 15368 raft_consensus.cc:493] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:43.843111 15368 raft_consensus.cc:3060] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:43.843444 14714 tablet_bootstrap.cc:492] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:43.838447 15299 rpcz_store.cc:276] 1211 08:57:42.505475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.505830 (+ 355us) service_pool.cc:225] Handling call
1211 08:57:43.610451 (+1104621us) ts_tablet_manager.cc:1151] Deleting tablet 4eec7694087145d8a49dd93904f95980
1211 08:57:43.610485 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.753762 (+143277us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54001243d4. stack: 00007efeecdf4083 00007efeecdf3f80 00007efeec969574 0000000000508744 00000000005086d9 00007efef3e3b5a1 00007efef3ed9fe2 00007efef42fc2ec 00007efef4f5331a 00007efef4f83f80 00007efef4f6a54d 00007efef4f6a4f9 00007efef4f6a489 00007efef4f6a451 00007efef4f6967d 00000000005aa2b4
1211 08:57:43.801780 (+ 48018us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.810190 (+ 8410us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:43.829609 (+ 19419us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":325721984,"tablet-delete.queue_time_us":1104129}
I20251211 08:57:43.842413 14722 consensus_queue.cc:260] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.847307 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 4eec7694087145d8a49dd93904f95980 (table default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]) successfully deleted
I20251211 08:57:43.850499 15368 raft_consensus.cc:515] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.854027 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 578404732eef4e51b66b6aebffa87cdb (table default.Table_53b4b18528d343018f204b9a3b0f9e63 [id=26c0a9d9b1f84372b18a54f045eb768c]). Expected version 2 got 0
I20251211 08:57:43.854574 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 41d6ef975a47489385791065bb8f9f46 (table default.Table_c9d62497a342488290749d4f7c02e748 [id=52ae4637827d424eb4d37362bf21cea2]). Expected version 2 got 0
I20251211 08:57:43.858994 15299 tablet_replica.cc:333] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:43.860793 15493 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:43.862468 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e813a4c73334c839bfe6af2119e259c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:43.862912 15500 mvcc.cc:204] Tried to move back new op lower bound from 7231256426580205568 to 7231256423485386752. Current Snapshot: MvccSnapshot[applied={T|T < 7231256426580205568}]
I20251211 08:57:43.859709 15299 raft_consensus.cc:2243] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:43.864173 15299 pending_rounds.cc:70] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:43.864377 15299 pending_rounds.cc:77] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256427824840704 op_type: NO_OP noop_request { }
I20251211 08:57:43.864705 15299 raft_consensus.cc:2889] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:43.864995 15299 raft_consensus.cc:2272] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:43.876789 15299 ts_tablet_manager.cc:1916] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:43.879529 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_cbfedb62a1204c96adec00226098c500" } new_table_name: "default.Table_d36a9570b7c0483a8373043af972d0cc" modify_external_catalogs: true
I20251211 08:57:43.881009 14714 tablet_bootstrap.cc:654] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:43.884295 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_e614e9972e50436aa5ad310e68a914fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:43.874533 14231 leader_election.cc:343] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: a6570281e1064c79b6e2297723f28658
W20251211 08:57:43.885957 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e614e9972e50436aa5ad310e68a914fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:43.890057 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6570281e1064c79b6e2297723f28658" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:43.889940 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_eacf82448cad4680a9e0ca3abe8ee5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:43.890491 14607 raft_consensus.cc:3060] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:43.891239 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eacf82448cad4680a9e0ca3abe8ee5bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:43.896250 15513 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:43.898216 15514 raft_consensus.cc:493] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:43.898602 15514 raft_consensus.cc:515] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.900736 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0e0ee99989d41c6af82e330ab8e353c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:43.901758 14544 leader_election.cc:343] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:43.911011 14231 leader_election.cc:343] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: a6570281e1064c79b6e2297723f28658
I20251211 08:57:43.918546 14607 raft_consensus.cc:2468] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:43.919952 14230 leader_election.cc:304] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:43.920715 15368 leader_election.cc:290] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:43.920734 15512 raft_consensus.cc:2804] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:43.924222 15512 raft_consensus.cc:697] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:43.924969 15512 consensus_queue.cc:237] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:43.932133 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_eb3b1d93a3ec4891a9d33b131661fd65" } modify_external_catalogs: true
I20251211 08:57:43.932557 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_eb3b1d93a3ec4891a9d33b131661fd65" } modify_external_catalogs: true
I20251211 08:57:43.936079 14722 ts_tablet_manager.cc:1434] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.097s user 0.007s sys 0.000s
I20251211 08:57:43.936086 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_917836f5cc9542c1a13cd99b4456b033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:43.938652 14722 tablet_bootstrap.cc:492] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:43.955240 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_c9d62497a342488290749d4f7c02e748" } new_table_name: "default.Table_846ceafaf2c74f698b8077c65736e2cf" modify_external_catalogs: true
I20251211 08:57:43.960327 15514 leader_election.cc:290] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:43.960443 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0e0ee99989d41c6af82e330ab8e353c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:43.961084 14445 raft_consensus.cc:2468] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:43.962756 14542 leader_election.cc:304] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:43.964558 15515 raft_consensus.cc:2804] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:43.964921 15515 raft_consensus.cc:493] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:43.965324 15515 raft_consensus.cc:3060] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:43.986013 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15595c5a432047de873c2647b78838d6 consensus state has no leader
I20251211 08:57:43.995380 14714 tablet_bootstrap.cc:492] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:43.995877 14714 ts_tablet_manager.cc:1403] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.153s user 0.011s sys 0.007s
W20251211 08:57:43.996194 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_917836f5cc9542c1a13cd99b4456b033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:44.001884 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ad600554c0946e0bcb0da6a42be2ffd with delete_type TABLET_DATA_DELETED (replaced by tablet 6874fb018bc844b3aeeab7db8fc53e9a) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:44.008944 14722 tablet_bootstrap.cc:654] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:44.014819 15299 ts_tablet_manager.cc:1929] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:44.015153 15299 log.cc:1199] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/f909d82e0136470a810e1063d6769868
I20251211 08:57:44.015825 15299 ts_tablet_manager.cc:1950] T f909d82e0136470a810e1063d6769868 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:44.017645 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 316) took 1490 ms. Trace:
I20251211 08:57:44.022863 14011 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_111bf58f55b84e8088d40992e55dd3f2" } modify_external_catalogs: true
I20251211 08:57:44.023365 14011 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_111bf58f55b84e8088d40992e55dd3f2" } modify_external_catalogs: true
I20251211 08:57:44.028050 14905 ts_tablet_manager.cc:1929] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:44.028410 14905 log.cc:1199] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/78c6a703e1e44f41931fe465def01bf6
I20251211 08:57:44.028981 14905 ts_tablet_manager.cc:1950] T 78c6a703e1e44f41931fe465def01bf6 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:44.032500 15520 tablet.cc:1722] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:44.034845 15030 tablet_bootstrap.cc:492] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:44.035182 15030 ts_tablet_manager.cc:1403] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.365s user 0.015s sys 0.005s
I20251211 08:57:44.037231 15030 raft_consensus.cc:359] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.037828 15030 raft_consensus.cc:385] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:44.038082 15030 raft_consensus.cc:740] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:44.038693 15030 consensus_queue.cc:260] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.043553 15515 raft_consensus.cc:515] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.029098 15493 tablet.cc:1722] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:44.044656 15299 rpcz_store.cc:276] 1211 08:57:42.527739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.527852 (+ 113us) service_pool.cc:225] Handling call
1211 08:57:43.849813 (+1321961us) ts_tablet_manager.cc:1151] Deleting tablet f909d82e0136470a810e1063d6769868
1211 08:57:43.858860 (+ 9047us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.932716 (+ 73856us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.014589 (+ 81873us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.017597 (+ 3008us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1321458}
I20251211 08:57:44.045794 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet f909d82e0136470a810e1063d6769868 (table default.Table_c15367b621c144348ec8511d2947fec6 [id=ea01e214ed314a26a0de20d34e37863b]) successfully deleted
I20251211 08:57:44.040086 14714 raft_consensus.cc:359] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.046461 14714 raft_consensus.cc:385] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:44.046737 14714 raft_consensus.cc:740] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:44.047498 14714 consensus_queue.cc:260] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.051517 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_034c882ef42046d58a86c1b90f5cdf85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:44.052024 15299 tablet_replica.cc:333] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:44.052789 15299 raft_consensus.cc:2243] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:44.053272 15299 raft_consensus.cc:2272] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:44.055560 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_034c882ef42046d58a86c1b90f5cdf85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:44.041627 15512 raft_consensus.cc:493] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:44.059226 15512 raft_consensus.cc:515] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.061445 15512 leader_election.cc:290] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:44.066017 14231 leader_election.cc:343] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 8ad600554c0946e0bcb0da6a42be2ffd
I20251211 08:57:44.066184 14714 ts_tablet_manager.cc:1434] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.070s user 0.006s sys 0.000s
I20251211 08:57:44.063324 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ad600554c0946e0bcb0da6a42be2ffd" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:44.010761 14904 ts_tablet_manager.cc:1929] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:44.066849 14904 log.cc:1199] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/12379255548f49dfad17f3840a0c86bb
I20251211 08:57:44.066895 14607 raft_consensus.cc:2468] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:44.067901 14230 leader_election.cc:304] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:44.068904 15512 raft_consensus.cc:2804] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:44.069178 15512 raft_consensus.cc:493] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:44.069445 15512 raft_consensus.cc:3060] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:44.070887 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0e0ee99989d41c6af82e330ab8e353c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:44.072113 14544 leader_election.cc:343] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:44.073926 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0e0ee99989d41c6af82e330ab8e353c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:44.074465 14445 raft_consensus.cc:3060] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:44.084692 15512 raft_consensus.cc:515] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:44.087591 14231 leader_election.cc:343] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 8ad600554c0946e0bcb0da6a42be2ffd
I20251211 08:57:44.088589 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 277) took 2014 ms. Trace:
I20251211 08:57:44.088891 14905 rpcz_store.cc:276] 1211 08:57:42.074614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.085164 (+ 10550us) service_pool.cc:225] Handling call
1211 08:57:43.748323 (+1663159us) ts_tablet_manager.cc:1151] Deleting tablet 78c6a703e1e44f41931fe465def01bf6
1211 08:57:43.748354 (+ 31us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.933021 (+184667us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.027835 (+ 94814us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.088532 (+ 60697us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1661868}
I20251211 08:57:44.089936 14722 tablet_bootstrap.cc:492] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:44.090027 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 275) took 2043 ms. Trace:
I20251211 08:57:44.090308 14722 ts_tablet_manager.cc:1403] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.154s user 0.009s sys 0.002s
I20251211 08:57:44.090266 14905 rpcz_store.cc:276] 1211 08:57:42.047309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.086920 (+ 39611us) service_pool.cc:225] Handling call
1211 08:57:44.089716 (+2002796us) ts_tablet_manager.cc:1151] Deleting tablet 2a1b011cb7c647f79bb16ce1c92bcff5
1211 08:57:44.089755 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.089982 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2002006}
I20251211 08:57:44.092024 14904 ts_tablet_manager.cc:1950] T 12379255548f49dfad17f3840a0c86bb P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:44.092693 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 275) took 1472 ms. Trace:
I20251211 08:57:44.092365 14722 raft_consensus.cc:359] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.092972 14904 rpcz_store.cc:276] 1211 08:57:42.620777 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.620882 (+ 105us) service_pool.cc:225] Handling call
1211 08:57:43.758932 (+1138050us) ts_tablet_manager.cc:1151] Deleting tablet 12379255548f49dfad17f3840a0c86bb
1211 08:57:43.758969 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:43.940291 (+181322us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.010582 (+ 70291us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.092651 (+ 82069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1137558}
I20251211 08:57:44.093559 14722 raft_consensus.cc:385] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:44.093743 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 78c6a703e1e44f41931fe465def01bf6 (table default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]) successfully deleted
I20251211 08:57:44.093812 14722 raft_consensus.cc:740] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:44.094094 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 282) took 1315 ms. Trace:
I20251211 08:57:44.094393 14722 consensus_queue.cc:260] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.094996 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 12379255548f49dfad17f3840a0c86bb (table default.Table_da9fb81b38a14423b6191db9f7a7c3df [id=e10eaa167f0a46c39674a5e2212f2226]) successfully deleted
I20251211 08:57:44.103626 14445 raft_consensus.cc:2468] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:44.106890 15299 ts_tablet_manager.cc:1916] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:44.109897 14714 tablet_bootstrap.cc:492] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:44.104730 14542 leader_election.cc:304] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:44.123539 15030 ts_tablet_manager.cc:1434] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.088s user 0.006s sys 0.000s
W20251211 08:57:44.123636 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a1b011cb7c647f79bb16ce1c92bcff5 already in progress: creating tablet
I20251211 08:57:44.125547 14722 ts_tablet_manager.cc:1434] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20251211 08:57:44.126225 15443 raft_consensus.cc:2804] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:44.126658 15443 raft_consensus.cc:697] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:44.127420 15443 consensus_queue.cc:237] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.094364 14904 rpcz_store.cc:276] 1211 08:57:42.778197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.778303 (+ 106us) service_pool.cc:225] Handling call
1211 08:57:44.093816 (+1315513us) ts_tablet_manager.cc:1151] Deleting tablet 2ed430a2c82d498a9c3ca116eaac154e
1211 08:57:44.093852 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.094049 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1315012}
I20251211 08:57:44.131654 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 284) took 1352 ms. Trace:
W20251211 08:57:44.131783 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 2ed430a2c82d498a9c3ca116eaac154e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ed430a2c82d498a9c3ca116eaac154e already in progress: creating tablet
I20251211 08:57:44.131911 14904 rpcz_store.cc:276] 1211 08:57:42.778770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.790884 (+ 12114us) service_pool.cc:225] Handling call
1211 08:57:44.131312 (+1340428us) ts_tablet_manager.cc:1151] Deleting tablet 98d14ba15dcd47f8b9e8035b3c7d477c
1211 08:57:44.131356 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.131612 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1339760}
I20251211 08:57:44.132668 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_bed4cef3d74f4af8acaa18fed506bdea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:44.133049 15512 leader_election.cc:290] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:44.133474 14722 tablet_bootstrap.cc:492] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:44.133988 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bed4cef3d74f4af8acaa18fed506bdea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:44.135162 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 98d14ba15dcd47f8b9e8035b3c7d477c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98d14ba15dcd47f8b9e8035b3c7d477c already in progress: creating tablet
I20251211 08:57:44.137871 14006 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 41d6ef975a47489385791065bb8f9f46 (table default.Table_846ceafaf2c74f698b8077c65736e2cf [id=52ae4637827d424eb4d37362bf21cea2]). Expected version 3 got 0
I20251211 08:57:44.141731 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ad600554c0946e0bcb0da6a42be2ffd" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:44.142211 14607 raft_consensus.cc:3060] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:44.145677 15030 tablet_bootstrap.cc:492] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:44.149927 14607 raft_consensus.cc:2468] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:44.151257 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 283) took 1372 ms. Trace:
I20251211 08:57:44.151566 14904 rpcz_store.cc:276] 1211 08:57:42.778473 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.795290 (+ 16817us) service_pool.cc:225] Handling call
1211 08:57:44.132617 (+1337327us) ts_tablet_manager.cc:1151] Deleting tablet f56fd93aeac6435ea84388ce05580113
1211 08:57:44.150854 (+ 18237us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.151205 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1336860}
I20251211 08:57:44.141820 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 276) took 2071 ms. Trace:
I20251211 08:57:44.152086 14230 leader_election.cc:304] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:44.152354 14905 rpcz_store.cc:276] 1211 08:57:42.070940 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.087981 (+ 17041us) service_pool.cc:225] Handling call
1211 08:57:44.123389 (+2035408us) ts_tablet_manager.cc:1151] Deleting tablet 570c7f08c46e4273bd38c32593dcf304
1211 08:57:44.141523 (+ 18134us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.141772 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2034624}
I20251211 08:57:44.153532 14905 tablet_replica.cc:333] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:57:44.153767 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 570c7f08c46e4273bd38c32593dcf304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570c7f08c46e4273bd38c32593dcf304 already in progress: creating tablet
I20251211 08:57:44.154206 14905 raft_consensus.cc:2243] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:44.154634 14905 raft_consensus.cc:2272] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:44.154911 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet f56fd93aeac6435ea84388ce05580113 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f56fd93aeac6435ea84388ce05580113 already in progress: creating tablet
I20251211 08:57:44.155854 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_5ceedd3100314612ab5b31cbcede0571"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:44.156335 14905 ts_tablet_manager.cc:1916] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:44.157421 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ceedd3100314612ab5b31cbcede0571 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:44.160580 15512 raft_consensus.cc:2804] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:44.161465 15512 raft_consensus.cc:697] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:44.162936 14904 tablet_replica.cc:333] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:44.165745 14904 raft_consensus.cc:2243] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:44.166391 14904 pending_rounds.cc:70] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
I20251211 08:57:44.166615 14904 pending_rounds.cc:77] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256428314443776 op_type: NO_OP noop_request { }
I20251211 08:57:44.166985 14904 raft_consensus.cc:2889] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:44.167304 14904 raft_consensus.cc:2272] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:44.170738 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 7f0cd7bb9fae4a2393578922fd359e6e (DEFAULT_TABLE table=default.Table_20ad075bb5ab4a5dadc32bcfe94b0322 [id=1e8b5ddeeea54c5f8e1951c184d77c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.172482 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f0cd7bb9fae4a2393578922fd359e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.162592 15512 consensus_queue.cc:237] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.105547 15515 leader_election.cc:290] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:44.180076 15030 tablet_bootstrap.cc:654] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:44.181613 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_624d973765514fb29d96a32ce2dff370" } new_table_name: "default.Table_9294d4662790426dbb71410c2c874bc0" modify_external_catalogs: true
I20251211 08:57:44.188899 14012 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 41d6ef975a47489385791065bb8f9f46 (table default.Table_846ceafaf2c74f698b8077c65736e2cf [id=52ae4637827d424eb4d37362bf21cea2]). Expected version 3 got 0
I20251211 08:57:44.192683 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_53b4b18528d343018f204b9a3b0f9e63" } new_table_name: "default.Table_27e68705f39e45fda963e581c94a0d45" modify_external_catalogs: true
I20251211 08:57:44.198269 14722 tablet_bootstrap.cc:654] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:44.199765 14904 ts_tablet_manager.cc:1916] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:44.201185 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 7f0cd7bb9fae4a2393578922fd359e6e (DEFAULT_TABLE table=default.Table_20ad075bb5ab4a5dadc32bcfe94b0322 [id=1e8b5ddeeea54c5f8e1951c184d77c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.202057 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f0cd7bb9fae4a2393578922fd359e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.217453 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 7f0cd7bb9fae4a2393578922fd359e6e (DEFAULT_TABLE table=default.Table_20ad075bb5ab4a5dadc32bcfe94b0322 [id=1e8b5ddeeea54c5f8e1951c184d77c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.218413 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f0cd7bb9fae4a2393578922fd359e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.218315 14268 tablet_service.cc:1505] Processing CreateTablet for tablet 785a5edbda984467b897ee1bbaf1430c (DEFAULT_TABLE table=default.Table_8d772a8a54e2416dacc1df571fe6042c [id=91b506cbcc1f43838defe979cb7ddd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.219221 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785a5edbda984467b897ee1bbaf1430c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.227372 14011 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 4c95565ce3954cc598d7cbd45321d7c9
I20251211 08:57:44.229674 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c95565ce3954cc598d7cbd45321d7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:44 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:44.230137 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c95565ce3954cc598d7cbd45321d7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:44 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:44.231017 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet ed0e90efe651499c929b1fb992712b96
I20251211 08:57:44.233240 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet ed0e90efe651499c929b1fb992712b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:44.234139 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet ed0e90efe651499c929b1fb992712b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:44.235237 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c95565ce3954cc598d7cbd45321d7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:44 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:44.235935 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet ed0e90efe651499c929b1fb992712b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:44.237864 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6f4cd1808b3d49929bca9414357073b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:44.239310 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f4cd1808b3d49929bca9414357073b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:44.241022 14714 tablet_bootstrap.cc:654] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:44.253645 15030 tablet_bootstrap.cc:492] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:44.254119 15030 ts_tablet_manager.cc:1403] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.000s
I20251211 08:57:44.257539 15030 raft_consensus.cc:359] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.258198 15030 raft_consensus.cc:385] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:44.258497 15030 raft_consensus.cc:740] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:44.259069 15030 consensus_queue.cc:260] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.264894 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 785a5edbda984467b897ee1bbaf1430c (DEFAULT_TABLE table=default.Table_8d772a8a54e2416dacc1df571fe6042c [id=91b506cbcc1f43838defe979cb7ddd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.265836 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785a5edbda984467b897ee1bbaf1430c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.265944 15030 ts_tablet_manager.cc:1434] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20251211 08:57:44.269315 14714 tablet_bootstrap.cc:492] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:44.270553 14714 ts_tablet_manager.cc:1403] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.161s user 0.008s sys 0.001s
I20251211 08:57:44.278381 14714 raft_consensus.cc:359] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.279765 14714 raft_consensus.cc:385] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:44.280805 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 07f983604bfa474f9c9ce0c982d738fc
I20251211 08:57:44.281883 14005 catalog_manager.cc:6724] ReplaceTablet: tablet 07f983604bfa474f9c9ce0c982d738fc deleted and replaced by tablet 960bc7a27fa04f2c9f398a67d2c93266
I20251211 08:57:44.280205 14010 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_53b4b18528d343018f204b9a3b0f9e63 [id=26c0a9d9b1f84372b18a54f045eb768c] alter complete (version 2)
I20251211 08:57:44.287654 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 07f983604bfa474f9c9ce0c982d738fc with delete_type TABLET_DATA_DELETED (replaced by tablet 960bc7a27fa04f2c9f398a67d2c93266) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:44.289552 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet 07f983604bfa474f9c9ce0c982d738fc with delete_type TABLET_DATA_DELETED (replaced by tablet 960bc7a27fa04f2c9f398a67d2c93266) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:44.291340 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6d419632b2a49e1975d4baa8ac85322 consensus state has no leader
I20251211 08:57:44.298183 15030 tablet_bootstrap.cc:492] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:44.305095 15030 tablet_bootstrap.cc:654] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:44.289009 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 07f983604bfa474f9c9ce0c982d738fc with delete_type TABLET_DATA_DELETED (replaced by tablet 960bc7a27fa04f2c9f398a67d2c93266) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:44.289182 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet f909d82e0136470a810e1063d6769868 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:44.307358 14714 raft_consensus.cc:740] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:44.317067 14905 ts_tablet_manager.cc:1929] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:44.317591 14905 log.cc:1199] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/a6570281e1064c79b6e2297723f28658
I20251211 08:57:44.318261 14905 ts_tablet_manager.cc:1950] T a6570281e1064c79b6e2297723f28658 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:44.319348 14722 tablet_bootstrap.cc:492] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:44.319711 14722 ts_tablet_manager.cc:1403] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.186s user 0.008s sys 0.009s
I20251211 08:57:44.319952 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 280) took 1985 ms. Trace:
I20251211 08:57:44.322249 14722 raft_consensus.cc:359] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.322856 14722 raft_consensus.cc:385] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:44.323136 14722 raft_consensus.cc:740] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:44.323622 14277 tablet_service.cc:1505] Processing CreateTablet for tablet b5bcc02258e04d8a8dd4a47c09fcbf47 (DEFAULT_TABLE table=default.Table_9bb3b4bd9d584bd6911f3b1cacc551df [id=c3336b33f67747db8d85c872faacad72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.323679 14722 consensus_queue.cc:260] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.324735 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bcc02258e04d8a8dd4a47c09fcbf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.325686 14722 ts_tablet_manager.cc:1434] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20251211 08:57:44.332482 14005 master_service.cc:954] ReplaceTablet: received request to replace tablet 5d8d84e594ea47f3bc9127cc587e8aa7 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:44.339694 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_bb1764135c31417987cdab15784dbea6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:44.341140 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb1764135c31417987cdab15784dbea6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:44.342142 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 785a5edbda984467b897ee1bbaf1430c (DEFAULT_TABLE table=default.Table_8d772a8a54e2416dacc1df571fe6042c [id=91b506cbcc1f43838defe979cb7ddd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.343199 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785a5edbda984467b897ee1bbaf1430c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.347595 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 90c8960c5bfa4a08905bd211654e4798 (table default.Table_d2367e8cf559401c8af423ce7425f5a8 [id=9f3e442a656a4a8080fbf847cb136b00]). Expected version 1 got 0
I20251211 08:57:44.347769 15517 raft_consensus.cc:493] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:44.348202 15517 raft_consensus.cc:515] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.349848 15517 leader_election.cc:290] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:44.350982 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41d6ef975a47489385791065bb8f9f46" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:44.350956 14904 ts_tablet_manager.cc:1929] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:44.352344 14904 log.cc:1199] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/a6570281e1064c79b6e2297723f28658
I20251211 08:57:44.352512 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet e961c5a3b5f640cbaea78b6185048d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:44.352919 14904 ts_tablet_manager.cc:1950] T a6570281e1064c79b6e2297723f28658 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:44.320235 14905 rpcz_store.cc:276] 1211 08:57:42.334674 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.334816 (+ 142us) service_pool.cc:225] Handling call
1211 08:57:44.153396 (+1818580us) ts_tablet_manager.cc:1151] Deleting tablet a6570281e1064c79b6e2297723f28658
1211 08:57:44.153430 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.231826 (+ 78396us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.316745 (+ 84919us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.319889 (+ 3144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1817980}
I20251211 08:57:44.354348 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_30906dd001234b1aa252daaa6df361e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:44.354938 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet a6570281e1064c79b6e2297723f28658 (table default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b [id=45f67c67257e428f8ed2f16898ad5fe6]) successfully deleted
W20251211 08:57:44.355726 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30906dd001234b1aa252daaa6df361e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:44.354756 14009 catalog_manager.cc:5654] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:44.357223 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 90c8960c5bfa4a08905bd211654e4798 (table default.Table_d2367e8cf559401c8af423ce7425f5a8 [id=9f3e442a656a4a8080fbf847cb136b00]). Expected version 1 got 0
I20251211 08:57:44.364526 14578 tablet_service.cc:1505] Processing CreateTablet for tablet b5bcc02258e04d8a8dd4a47c09fcbf47 (DEFAULT_TABLE table=default.Table_9bb3b4bd9d584bd6911f3b1cacc551df [id=c3336b33f67747db8d85c872faacad72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.365450 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bcc02258e04d8a8dd4a47c09fcbf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.367705 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 844a8e38a2354c839c429d635eaf46a1 (DEFAULT_TABLE table=default.Table_0e8a3c6f663b4834983142a353b03e40 [id=453b9f570a2f4e55a7b54ffc1c719198]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.367892 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 844a8e38a2354c839c429d635eaf46a1 (DEFAULT_TABLE table=default.Table_0e8a3c6f663b4834983142a353b03e40 [id=453b9f570a2f4e55a7b54ffc1c719198]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.368564 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 844a8e38a2354c839c429d635eaf46a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.368755 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 844a8e38a2354c839c429d635eaf46a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.354681 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41d6ef975a47489385791065bb8f9f46" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:44.365427 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 844a8e38a2354c839c429d635eaf46a1 (DEFAULT_TABLE table=default.Table_0e8a3c6f663b4834983142a353b03e40 [id=453b9f570a2f4e55a7b54ffc1c719198]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.382236 15520 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:44.383118 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 844a8e38a2354c839c429d635eaf46a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.379817 15493 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:44.384084 14722 tablet_bootstrap.cc:492] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:44.387286 14231 leader_election.cc:304] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:44.388490 15517 raft_consensus.cc:2749] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:44.397893 14714 consensus_queue.cc:260] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.399703 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6bd043cbc5024c4c9881e15f002ce3cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:44.401167 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bd043cbc5024c4c9881e15f002ce3cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:44.403118 14905 tablet_replica.cc:333] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:44.404215 14905 raft_consensus.cc:2243] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:44.405339 14905 pending_rounds.cc:70] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:57:44.405630 14905 pending_rounds.cc:77] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256416673222656 op_type: NO_OP noop_request { }
I20251211 08:57:44.406605 14905 raft_consensus.cc:2889] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:44.407086 14905 raft_consensus.cc:2272] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:44.403234 15122 raft_consensus.cc:493] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:44.408339 15122 raft_consensus.cc:515] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.408975 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 289) took 1497 ms. Trace:
I20251211 08:57:44.409251 14904 rpcz_store.cc:276] 1211 08:57:42.911179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.911269 (+ 90us) service_pool.cc:225] Handling call
1211 08:57:44.152255 (+1240986us) ts_tablet_manager.cc:1151] Deleting tablet a6570281e1064c79b6e2297723f28658
1211 08:57:44.162814 (+ 10559us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.242698 (+ 79884us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.329929 (+ 87231us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.408921 (+ 78992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1240407}
I20251211 08:57:44.409873 15122 leader_election.cc:290] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:44.410523 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 290) took 1375 ms. Trace:
I20251211 08:57:44.410789 14904 rpcz_store.cc:276] 1211 08:57:43.035500 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.035620 (+ 120us) service_pool.cc:225] Handling call
1211 08:57:44.410257 (+1374637us) ts_tablet_manager.cc:1151] Deleting tablet 7af7c9ab27a64a1681f9874a922e986f
1211 08:57:44.410292 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.410481 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1374153}
I20251211 08:57:44.411187 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e813a4c73334c839bfe6af2119e259c" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:44.412195 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet a6570281e1064c79b6e2297723f28658 (table default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b [id=45f67c67257e428f8ed2f16898ad5fe6]) successfully deleted
W20251211 08:57:44.412814 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 7af7c9ab27a64a1681f9874a922e986f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7af7c9ab27a64a1681f9874a922e986f already in progress: creating tablet
W20251211 08:57:44.412809 14382 leader_election.cc:343] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:44.413540 14904 tablet_replica.cc:333] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:44.413826 14722 tablet_bootstrap.cc:654] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:44.413980 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e813a4c73334c839bfe6af2119e259c" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:44.414217 14904 raft_consensus.cc:2243] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:44.415087 14904 raft_consensus.cc:2272] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:44.415156 14385 leader_election.cc:343] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:44.415464 14385 leader_election.cc:304] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:44.416682 14904 ts_tablet_manager.cc:1916] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:44.417097 15122 raft_consensus.cc:2749] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:44.421996 14714 ts_tablet_manager.cc:1434] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.148s user 0.005s sys 0.000s
I20251211 08:57:44.439088 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 5d8d84e594ea47f3bc9127cc587e8aa7
I20251211 08:57:44.439024 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f2a748aebc4d47738c1758ed749c0f98" } modify_external_catalogs: true
I20251211 08:57:44.439487 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f2a748aebc4d47738c1758ed749c0f98" } modify_external_catalogs: true
I20251211 08:57:44.440420 14005 catalog_manager.cc:6724] ReplaceTablet: tablet 5d8d84e594ea47f3bc9127cc587e8aa7 deleted and replaced by tablet e62ae5b540aa4680b4d18fee29c39e64
I20251211 08:57:44.441983 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 with delete_type TABLET_DATA_DELETED (replaced by tablet e62ae5b540aa4680b4d18fee29c39e64) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:44.443122 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9ec80b196de446bf8a12c0ae014f783e" } modify_external_catalogs: true
I20251211 08:57:44.443758 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9ec80b196de446bf8a12c0ae014f783e" } modify_external_catalogs: true
I20251211 08:57:44.454134 14010 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_846ceafaf2c74f698b8077c65736e2cf [id=52ae4637827d424eb4d37362bf21cea2] alter complete (version 3)
I20251211 08:57:44.381075 14420 tablet_service.cc:1505] Processing CreateTablet for tablet b5bcc02258e04d8a8dd4a47c09fcbf47 (DEFAULT_TABLE table=default.Table_9bb3b4bd9d584bd6911f3b1cacc551df [id=c3336b33f67747db8d85c872faacad72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.455175 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 with delete_type TABLET_DATA_DELETED (replaced by tablet e62ae5b540aa4680b4d18fee29c39e64) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:44.460956 14714 tablet_bootstrap.cc:492] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:44.466058 15030 tablet_bootstrap.cc:492] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:44.470538 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bcc02258e04d8a8dd4a47c09fcbf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.481616 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 485255e09a06424ea37e9a03023ddcb8 (DEFAULT_TABLE table=default.Table_0746d662358e4d9681b7d0a3529fb584 [id=a3378803e96e421e817203890029987f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.489847 15030 ts_tablet_manager.cc:1403] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.192s user 0.006s sys 0.003s
I20251211 08:57:44.496707 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485255e09a06424ea37e9a03023ddcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.472055 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 485255e09a06424ea37e9a03023ddcb8 (DEFAULT_TABLE table=default.Table_0746d662358e4d9681b7d0a3529fb584 [id=a3378803e96e421e817203890029987f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.476356 14905 ts_tablet_manager.cc:1916] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:44.499106 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485255e09a06424ea37e9a03023ddcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.502882 14011 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ccb3af786cad4564b032d78520da1cc3" } new_table_name: "default.Table_1eb40c19ca894f47ba26da25462c5c3a" modify_external_catalogs: true
W20251211 08:57:44.506735 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet c6d419632b2a49e1975d4baa8ac85322 (table default.Table_9294d4662790426dbb71410c2c874bc0 [id=eec20447308f431092b2b3736ae03d47], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c6d419632b2a49e1975d4baa8ac85322 consensus state has no leader
I20251211 08:57:44.508045 15122 raft_consensus.cc:493] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:44.508423 15122 raft_consensus.cc:515] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.509914 15122 leader_election.cc:290] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:44.511204 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c8960c5bfa4a08905bd211654e4798" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:44.511981 14607 raft_consensus.cc:2468] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:44.513829 14382 leader_election.cc:304] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:44.514988 15122 raft_consensus.cc:2804] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:44.515380 15122 raft_consensus.cc:493] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:44.515707 15122 raft_consensus.cc:3060] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:44.516397 15299 ts_tablet_manager.cc:1929] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:44.516724 15299 log.cc:1199] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/12379255548f49dfad17f3840a0c86bb
I20251211 08:57:44.518445 14428 tablet_service.cc:1505] Processing CreateTablet for tablet f2ee6cab1d1742e08323842ad9c69b81 (DEFAULT_TABLE table=default.Table_5bb3e684c5f445188aa010ca4b32db87 [id=c57ca9997a5d4a3cac6111fc655c82ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.519698 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ee6cab1d1742e08323842ad9c69b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.500880 15030 raft_consensus.cc:359] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.522159 15030 raft_consensus.cc:385] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:44.522387 15030 raft_consensus.cc:740] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:44.525377 15299 ts_tablet_manager.cc:1950] T 12379255548f49dfad17f3840a0c86bb P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:44.529042 14578 tablet_service.cc:1505] Processing CreateTablet for tablet f2ee6cab1d1742e08323842ad9c69b81 (DEFAULT_TABLE table=default.Table_5bb3e684c5f445188aa010ca4b32db87 [id=c57ca9997a5d4a3cac6111fc655c82ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.529567 14714 tablet_bootstrap.cc:654] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:44.530285 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ee6cab1d1742e08323842ad9c69b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.530221 15030 consensus_queue.cc:260] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.532207 15030 ts_tablet_manager.cc:1434] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.036s user 0.004s sys 0.000s
I20251211 08:57:44.535441 14427 tablet_service.cc:1505] Processing CreateTablet for tablet f3cf2b4d14ba45d79e52b6fdb0497d58 (DEFAULT_TABLE table=default.Table_74345a98ec224b2398937ca22336051b [id=945a9eb17575451192d09cf17a00a46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.527160 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 318) took 1905 ms. Trace:
I20251211 08:57:44.536311 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3cf2b4d14ba45d79e52b6fdb0497d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.536250 14586 tablet_service.cc:1505] Processing CreateTablet for tablet f3cf2b4d14ba45d79e52b6fdb0497d58 (DEFAULT_TABLE table=default.Table_74345a98ec224b2398937ca22336051b [id=945a9eb17575451192d09cf17a00a46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.536280 15299 rpcz_store.cc:276] 1211 08:57:42.621679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.621796 (+ 117us) service_pool.cc:225] Handling call
1211 08:57:44.051902 (+1430106us) ts_tablet_manager.cc:1151] Deleting tablet 12379255548f49dfad17f3840a0c86bb
1211 08:57:44.051937 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.364778 (+312841us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.516208 (+151430us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.527109 (+ 10901us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429664}
I20251211 08:57:44.539647 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 12379255548f49dfad17f3840a0c86bb (table default.Table_da9fb81b38a14423b6191db9f7a7c3df [id=e10eaa167f0a46c39674a5e2212f2226]) successfully deleted
I20251211 08:57:44.546051 14275 tablet_service.cc:1505] Processing CreateTablet for tablet f3cf2b4d14ba45d79e52b6fdb0497d58 (DEFAULT_TABLE table=default.Table_74345a98ec224b2398937ca22336051b [id=945a9eb17575451192d09cf17a00a46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.548524 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 96aee3525f2b466a9f19992a8795861d (DEFAULT_TABLE table=default.Table_b56b94bf721941859de71bcabd9b8b5c [id=535810711af54f4d9072471c0e7f993b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.550128 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 327) took 1767 ms. Trace:
I20251211 08:57:44.550400 15299 rpcz_store.cc:276] 1211 08:57:42.782645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.782743 (+ 98us) service_pool.cc:225] Handling call
1211 08:57:44.537447 (+1754704us) ts_tablet_manager.cc:1151] Deleting tablet f56fd93aeac6435ea84388ce05580113
1211 08:57:44.537486 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.550082 (+ 12596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1751900}
I20251211 08:57:44.551204 14268 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 with delete_type TABLET_DATA_DELETED (replaced by tablet e62ae5b540aa4680b4d18fee29c39e64) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:44.551528 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 328) took 1768 ms. Trace:
I20251211 08:57:44.551231 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3cf2b4d14ba45d79e52b6fdb0497d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.551867 15299 rpcz_store.cc:276] 1211 08:57:42.782941 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.788928 (+ 5987us) service_pool.cc:225] Handling call
1211 08:57:44.551204 (+1762276us) ts_tablet_manager.cc:1151] Deleting tablet 98d14ba15dcd47f8b9e8035b3c7d477c
1211 08:57:44.551242 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.551468 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1761336}
I20251211 08:57:44.552909 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 326) took 1770 ms. Trace:
I20251211 08:57:44.552893 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 485255e09a06424ea37e9a03023ddcb8 (DEFAULT_TABLE table=default.Table_0746d662358e4d9681b7d0a3529fb584 [id=a3378803e96e421e817203890029987f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.553160 15299 rpcz_store.cc:276] 1211 08:57:42.782403 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.802844 (+ 20441us) service_pool.cc:225] Handling call
1211 08:57:44.552600 (+1749756us) ts_tablet_manager.cc:1151] Deleting tablet 2ed430a2c82d498a9c3ca116eaac154e
1211 08:57:44.552629 (+ 29us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.552869 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1748232}
I20251211 08:57:44.553862 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485255e09a06424ea37e9a03023ddcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.554003 15299 tablet_replica.cc:333] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:44.555160 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96aee3525f2b466a9f19992a8795861d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.555747 14266 tablet_service.cc:1505] Processing CreateTablet for tablet f2ee6cab1d1742e08323842ad9c69b81 (DEFAULT_TABLE table=default.Table_5bb3e684c5f445188aa010ca4b32db87 [id=c57ca9997a5d4a3cac6111fc655c82ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.556917 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ee6cab1d1742e08323842ad9c69b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.563143 15512 raft_consensus.cc:493] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:44.563747 15512 raft_consensus.cc:515] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.554675 15299 raft_consensus.cc:2243] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:44.565251 15299 raft_consensus.cc:2272] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:44.566171 15512 leader_election.cc:290] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:44.568058 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c8960c5bfa4a08905bd211654e4798" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:44.568776 14722 tablet_bootstrap.cc:492] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:44.569295 14722 ts_tablet_manager.cc:1403] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.185s user 0.008s sys 0.006s
I20251211 08:57:44.571220 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b462cf21e3420c8e323a4ac4f04abe with delete_type TABLET_DATA_DELETED (replaced by tablet 2ed430a2c82d498a9c3ca116eaac154e) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:44.572275 14298 raft_consensus.cc:3060] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:44.571697 14722 raft_consensus.cc:359] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.573065 14722 raft_consensus.cc:385] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:44.573318 14722 raft_consensus.cc:740] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:44.573880 14722 consensus_queue.cc:260] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:44.577186 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f56fd93aeac6435ea84388ce05580113 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f56fd93aeac6435ea84388ce05580113 already in progress: creating tablet
W20251211 08:57:44.577797 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 98d14ba15dcd47f8b9e8035b3c7d477c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98d14ba15dcd47f8b9e8035b3c7d477c already in progress: creating tablet
W20251211 08:57:44.577961 14385 leader_election.cc:343] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:44.578298 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 2ed430a2c82d498a9c3ca116eaac154e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ed430a2c82d498a9c3ca116eaac154e already in progress: creating tablet
I20251211 08:57:44.580605 14722 ts_tablet_manager.cc:1434] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20251211 08:57:44.582535 15299 ts_tablet_manager.cc:1916] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:44.584380 14422 tablet_service.cc:1505] Processing CreateTablet for tablet 96aee3525f2b466a9f19992a8795861d (DEFAULT_TABLE table=default.Table_b56b94bf721941859de71bcabd9b8b5c [id=535810711af54f4d9072471c0e7f993b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.585213 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96aee3525f2b466a9f19992a8795861d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:44.588593 14231 leader_election.cc:343] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 174fefd70bcc41418a77a5716f97cd5f
I20251211 08:57:44.591866 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 96aee3525f2b466a9f19992a8795861d (DEFAULT_TABLE table=default.Table_b56b94bf721941859de71bcabd9b8b5c [id=535810711af54f4d9072471c0e7f993b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.593174 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 58d5ded96b9149f9ab0d9ea70e376af1 (DEFAULT_TABLE table=default.Table_873e2f0afae049eb85e3652c71880de5 [id=a4ac86adaf1b47739cebde39e1197382]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.609342 15030 tablet_bootstrap.cc:492] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:44.611282 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3cf2b4d14ba45d79e52b6fdb0497d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.612085 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96aee3525f2b466a9f19992a8795861d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.617142 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 58d5ded96b9149f9ab0d9ea70e376af1 (DEFAULT_TABLE table=default.Table_873e2f0afae049eb85e3652c71880de5 [id=a4ac86adaf1b47739cebde39e1197382]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.618055 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d5ded96b9149f9ab0d9ea70e376af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.619683 14268 tablet_service.cc:1505] Processing CreateTablet for tablet da124c4afb5a402fb212d0ea95e0b47e (DEFAULT_TABLE table=default.Table_2bcb516a7ea04fe7be2fd23eb963dd27 [id=ef1bc4f02c544473aef5e896740751d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.623507 14722 tablet_bootstrap.cc:492] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:44.630861 14299 raft_consensus.cc:3060] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:44.632995 15030 tablet_bootstrap.cc:654] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:44.641008 14424 tablet_service.cc:1505] Processing CreateTablet for tablet da124c4afb5a402fb212d0ea95e0b47e (DEFAULT_TABLE table=default.Table_2bcb516a7ea04fe7be2fd23eb963dd27 [id=ef1bc4f02c544473aef5e896740751d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.641413 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 58d5ded96b9149f9ab0d9ea70e376af1 (DEFAULT_TABLE table=default.Table_873e2f0afae049eb85e3652c71880de5 [id=a4ac86adaf1b47739cebde39e1197382]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.641862 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da124c4afb5a402fb212d0ea95e0b47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.642385 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d5ded96b9149f9ab0d9ea70e376af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.643465 14580 tablet_service.cc:1505] Processing CreateTablet for tablet da124c4afb5a402fb212d0ea95e0b47e (DEFAULT_TABLE table=default.Table_2bcb516a7ea04fe7be2fd23eb963dd27 [id=ef1bc4f02c544473aef5e896740751d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.644385 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da124c4afb5a402fb212d0ea95e0b47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.644922 14265 tablet_service.cc:1505] Processing CreateTablet for tablet 3838fd220d034b3595f351b906188339 (DEFAULT_TABLE table=default.Table_f0e893169e89418ab044ae00cdb78176 [id=6ef69beb843946d080b3f6784f0e3af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.668155 14416 tablet_service.cc:1505] Processing CreateTablet for tablet 3838fd220d034b3595f351b906188339 (DEFAULT_TABLE table=default.Table_f0e893169e89418ab044ae00cdb78176 [id=6ef69beb843946d080b3f6784f0e3af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.669147 14416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3838fd220d034b3595f351b906188339. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.678391 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 3838fd220d034b3595f351b906188339 (DEFAULT_TABLE table=default.Table_f0e893169e89418ab044ae00cdb78176 [id=6ef69beb843946d080b3f6784f0e3af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.682886 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3838fd220d034b3595f351b906188339. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.666725 14904 ts_tablet_manager.cc:1929] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251211 08:57:44.676267 14269 tablet_service.cc:1505] Processing CreateTablet for tablet 8d39234b58734161b3bcfb67d0df1bd4 (DEFAULT_TABLE table=default.Table_a4bc9b5633444f8587bf740793fd55a2 [id=1a961233d708456bb4cf8d3fc39dc705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.687333 14904 log.cc:1199] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/593df63696a04f85b00c301d93748e80
I20251211 08:57:44.679178 14271 tablet_service.cc:1505] Processing CreateTablet for tablet ca5eebf2ce3848c79ee2fc21ad2b1486 (DEFAULT_TABLE table=default.Table_064b00aac2ff440891fc0f5f0143a573 [id=5ea14e5cfb344904a01908d23ddf53c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.684003 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da124c4afb5a402fb212d0ea95e0b47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.688279 14904 ts_tablet_manager.cc:1950] T 593df63696a04f85b00c301d93748e80 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:44.679759 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 8d39234b58734161b3bcfb67d0df1bd4 (DEFAULT_TABLE table=default.Table_a4bc9b5633444f8587bf740793fd55a2 [id=1a961233d708456bb4cf8d3fc39dc705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.688808 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3838fd220d034b3595f351b906188339. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.689323 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d39234b58734161b3bcfb67d0df1bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.694145 15515 raft_consensus.cc:493] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:44.694530 15515 raft_consensus.cc:515] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.694988 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d5ded96b9149f9ab0d9ea70e376af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.695515 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 8d39234b58734161b3bcfb67d0df1bd4 (DEFAULT_TABLE table=default.Table_a4bc9b5633444f8587bf740793fd55a2 [id=1a961233d708456bb4cf8d3fc39dc705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.696159 15515 leader_election.cc:290] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:44.696730 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d39234b58734161b3bcfb67d0df1bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.697738 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5b462cf21e3420c8e323a4ac4f04abe" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:44.698220 14714 tablet_bootstrap.cc:492] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
W20251211 08:57:44.698973 14542 leader_election.cc:343] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: Tablet not RUNNING: STOPPED
I20251211 08:57:44.699199 14714 ts_tablet_manager.cc:1403] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.238s user 0.007s sys 0.007s
I20251211 08:57:44.699818 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5b462cf21e3420c8e323a4ac4f04abe" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:44.700937 14544 leader_election.cc:343] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:44.701237 14544 leader_election.cc:304] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:44.701547 14714 raft_consensus.cc:359] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.701983 15515 raft_consensus.cc:2749] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:44.702040 14714 raft_consensus.cc:385] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:44.703460 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d39234b58734161b3bcfb67d0df1bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.706753 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5eebf2ce3848c79ee2fc21ad2b1486. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.712553 14278 tablet_service.cc:1505] Processing CreateTablet for tablet ee50f29bfe674611a834d84a7c4e9070 (DEFAULT_TABLE table=default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.713546 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee50f29bfe674611a834d84a7c4e9070. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.715699 14417 tablet_service.cc:1505] Processing CreateTablet for tablet ca5eebf2ce3848c79ee2fc21ad2b1486 (DEFAULT_TABLE table=default.Table_064b00aac2ff440891fc0f5f0143a573 [id=5ea14e5cfb344904a01908d23ddf53c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.716749 14417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5eebf2ce3848c79ee2fc21ad2b1486. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.724566 14418 tablet_service.cc:1505] Processing CreateTablet for tablet ee50f29bfe674611a834d84a7c4e9070 (DEFAULT_TABLE table=default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.725610 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee50f29bfe674611a834d84a7c4e9070. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.732007 15030 tablet_bootstrap.cc:492] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:44.732376 15030 ts_tablet_manager.cc:1403] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.123s user 0.009s sys 0.000s
I20251211 08:57:44.740403 15299 ts_tablet_manager.cc:1929] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:44.744097 15299 log.cc:1199] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/f5b462cf21e3420c8e323a4ac4f04abe
I20251211 08:57:44.744181 14587 tablet_service.cc:1505] Processing CreateTablet for tablet ee50f29bfe674611a834d84a7c4e9070 (DEFAULT_TABLE table=default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.745157 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee50f29bfe674611a834d84a7c4e9070. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.749761 14275 tablet_service.cc:1505] Processing CreateTablet for tablet f2dc6cb4efc744e6904728ad120df934 (DEFAULT_TABLE table=default.Table_da9fb81b38a14423b6191db9f7a7c3df [id=e10eaa167f0a46c39674a5e2212f2226]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.750813 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2dc6cb4efc744e6904728ad120df934. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.752403 15299 ts_tablet_manager.cc:1950] T f5b462cf21e3420c8e323a4ac4f04abe P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:44.754416 14722 tablet_bootstrap.cc:700] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b: Time spent opening tablet: real 0.124s user 0.003s sys 0.000s
I20251211 08:57:44.755084 14722 tablet_bootstrap.cc:654] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:44.758061 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 291) took 1640 ms. Trace:
I20251211 08:57:44.758415 14904 rpcz_store.cc:276] 1211 08:57:43.117885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.118003 (+ 118us) service_pool.cc:225] Handling call
1211 08:57:44.413426 (+1295423us) ts_tablet_manager.cc:1151] Deleting tablet 593df63696a04f85b00c301d93748e80
1211 08:57:44.413457 (+ 31us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.607235 (+193778us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.666421 (+ 59186us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.757958 (+ 91537us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1294920}
I20251211 08:57:44.759178 14426 tablet_service.cc:1505] Processing CreateTablet for tablet f2dc6cb4efc744e6904728ad120df934 (DEFAULT_TABLE table=default.Table_da9fb81b38a14423b6191db9f7a7c3df [id=e10eaa167f0a46c39674a5e2212f2226]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.759749 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 593df63696a04f85b00c301d93748e80 (table default.Table_da883b3cac244bc0aee81f7c17ac81a6 [id=ab93470ba8f54070be718c240d49aa13]) successfully deleted
I20251211 08:57:44.760744 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 294) took 1479 ms. Trace:
I20251211 08:57:44.761019 14904 rpcz_store.cc:276] 1211 08:57:43.281702 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.281794 (+ 92us) service_pool.cc:225] Handling call
1211 08:57:44.760424 (+1478630us) ts_tablet_manager.cc:1151] Deleting tablet dd14de369e084828afb8a924c46c649b
1211 08:57:44.760480 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.760704 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1478122}
W20251211 08:57:44.762006 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet dd14de369e084828afb8a924c46c649b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd14de369e084828afb8a924c46c649b already in progress: creating tablet
I20251211 08:57:44.762748 14904 tablet_replica.cc:333] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:44.762145 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2dc6cb4efc744e6904728ad120df934. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.762844 15122 raft_consensus.cc:515] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.763540 14904 raft_consensus.cc:2243] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:44.763994 14904 raft_consensus.cc:2272] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:44.765542 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c8960c5bfa4a08905bd211654e4798" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:44.765947 14904 ts_tablet_manager.cc:1916] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:44.766520 14385 leader_election.cc:343] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:44.767649 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c8960c5bfa4a08905bd211654e4798" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:44.768178 14605 raft_consensus.cc:3060] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:44.771003 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 330) took 1912 ms. Trace:
I20251211 08:57:44.771293 15299 rpcz_store.cc:276] 1211 08:57:42.858292 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.872405 (+ 14113us) service_pool.cc:225] Handling call
1211 08:57:44.553875 (+1681470us) ts_tablet_manager.cc:1151] Deleting tablet f5b462cf21e3420c8e323a4ac4f04abe
1211 08:57:44.553911 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.678906 (+124995us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.740205 (+ 61299us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.770953 (+ 30748us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1680846}
I20251211 08:57:44.772289 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet f5b462cf21e3420c8e323a4ac4f04abe (table default.Table_e444843f69064fbb9c2a6d0671fd9a2a [id=6c43f429406443d7b5834905929dcbd4]) successfully deleted
I20251211 08:57:44.774374 14605 raft_consensus.cc:2468] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:44.775640 14576 tablet_service.cc:1505] Processing CreateTablet for tablet f2dc6cb4efc744e6904728ad120df934 (DEFAULT_TABLE table=default.Table_da9fb81b38a14423b6191db9f7a7c3df [id=e10eaa167f0a46c39674a5e2212f2226]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.776648 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2dc6cb4efc744e6904728ad120df934. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.784083 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 333) took 1756 ms. Trace:
I20251211 08:57:44.784453 14584 tablet_service.cc:1505] Processing CreateTablet for tablet ca5eebf2ce3848c79ee2fc21ad2b1486 (DEFAULT_TABLE table=default.Table_064b00aac2ff440891fc0f5f0143a573 [id=5ea14e5cfb344904a01908d23ddf53c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.785064 14296 raft_consensus.cc:3060] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:44.785876 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5eebf2ce3848c79ee2fc21ad2b1486. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.734622 15030 raft_consensus.cc:359] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.787310 15030 raft_consensus.cc:385] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:44.787688 15030 raft_consensus.cc:740] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:44.784360 15299 rpcz_store.cc:276] 1211 08:57:43.027508 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.027630 (+ 122us) service_pool.cc:225] Handling call
1211 08:57:44.783749 (+1756119us) ts_tablet_manager.cc:1151] Deleting tablet 7af7c9ab27a64a1681f9874a922e986f
1211 08:57:44.783801 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.784036 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1755594}
I20251211 08:57:44.791651 14273 tablet_service.cc:1505] Processing CreateTablet for tablet c6d419632b2a49e1975d4baa8ac85322 (DEFAULT_TABLE table=default.Table_9294d4662790426dbb71410c2c874bc0 [id=eec20447308f431092b2b3736ae03d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.702231 14714 raft_consensus.cc:740] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:44.792619 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d419632b2a49e1975d4baa8ac85322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.792505 14714 consensus_queue.cc:260] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.793427 14580 tablet_service.cc:1505] Processing CreateTablet for tablet c6d419632b2a49e1975d4baa8ac85322 (DEFAULT_TABLE table=default.Table_9294d4662790426dbb71410c2c874bc0 [id=eec20447308f431092b2b3736ae03d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.797140 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 9821822304c948f3bfcf5d52901371a4 (DEFAULT_TABLE table=default.Table_4e15a6b7343e46c491019358d60e3589 [id=f53f94afd6924a859e896f3974c4c5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.798219 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9821822304c948f3bfcf5d52901371a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.799223 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d419632b2a49e1975d4baa8ac85322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.807379 14382 leader_election.cc:304] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:44.808609 15030 consensus_queue.cc:260] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:44.811092 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 7af7c9ab27a64a1681f9874a922e986f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7af7c9ab27a64a1681f9874a922e986f already in progress: creating tablet
I20251211 08:57:44.816742 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 02c7fbd61b2e42efaa0ae06ed1b1091c (DEFAULT_TABLE table=default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.817463 15030 ts_tablet_manager.cc:1434] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.085s user 0.006s sys 0.000s
I20251211 08:57:44.817744 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c7fbd61b2e42efaa0ae06ed1b1091c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.822891 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "174fefd70bcc41418a77a5716f97cd5f" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:44.824522 14230 leader_election.cc:343] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: Tablet not RUNNING: STOPPED
I20251211 08:57:44.824882 14230 leader_election.cc:304] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
W20251211 08:57:44.825340 14230 raft_consensus.cc:2686] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251211 08:57:44.826592 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 02c7fbd61b2e42efaa0ae06ed1b1091c (DEFAULT_TABLE table=default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.827646 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c7fbd61b2e42efaa0ae06ed1b1091c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.834347 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 9821822304c948f3bfcf5d52901371a4 (DEFAULT_TABLE table=default.Table_4e15a6b7343e46c491019358d60e3589 [id=f53f94afd6924a859e896f3974c4c5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.835301 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9821822304c948f3bfcf5d52901371a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.837651 15030 tablet_bootstrap.cc:492] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:44.837913 15122 leader_election.cc:290] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:44.838479 15122 raft_consensus.cc:2804] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:44.847864 15512 raft_consensus.cc:493] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5e624bcfd154cd586ef1dedb9940063)
I20251211 08:57:44.848358 15512 raft_consensus.cc:515] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.849030 15030 tablet_bootstrap.cc:654] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:44.853147 15512 leader_election.cc:290] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:44.854653 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e961c5a3b5f640cbaea78b6185048d04" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:44.855006 15540 tablet.cc:1722] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:44.862404 14271 tablet_service.cc:1505] Processing CreateTablet for tablet 01310095bca4479da24185576091b748 (DEFAULT_TABLE table=default.Table_9ec80b196de446bf8a12c0ae014f783e [id=ee66bcc72bcf4627b996467b0825b1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.863672 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01310095bca4479da24185576091b748. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:44.866851 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_74345a98ec224b2398937ca22336051b" } new_table_name: "default.Table_d3f2e5059297427fbaaefd79504a2e2f" modify_external_catalogs: true
I20251211 08:57:44.869292 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_8f9a86bf44d14c63af54ebaca1cfbb67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:44.870553 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f9a86bf44d14c63af54ebaca1cfbb67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:44.872730 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7558ae666eb04c7b96e3d1151f22fbe6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:44.873632 15122 raft_consensus.cc:697] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
W20251211 08:57:44.873914 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7558ae666eb04c7b96e3d1151f22fbe6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:44.874955 15122 consensus_queue.cc:237] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.875195 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2839940578ba47e88ac23a1850c356ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:44.876296 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2839940578ba47e88ac23a1850c356ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:44.877342 14009 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 578404732eef4e51b66b6aebffa87cdb (table default.Table_27e68705f39e45fda963e581c94a0d45 [id=26c0a9d9b1f84372b18a54f045eb768c]). Expected version 3 got 0
I20251211 08:57:44.880995 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 90c8960c5bfa4a08905bd211654e4798 (table default.Table_d2367e8cf559401c8af423ce7425f5a8 [id=9f3e442a656a4a8080fbf847cb136b00]). Expected version 1 got 0
I20251211 08:57:44.889367 15440 raft_consensus.cc:493] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:44.889735 15440 raft_consensus.cc:515] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.891333 15440 leader_election.cc:290] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:44.892179 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e813a4c73334c839bfe6af2119e259c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:44.893241 14544 leader_election.cc:343] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:44.903218 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 4631dd0bc0c144e69a31de80de39a76a
I20251211 08:57:44.905117 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_e1fb587d58c340da84a5c8f0e0c1d7ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:44.911054 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 4631dd0bc0c144e69a31de80de39a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:44 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:44.906733 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1fb587d58c340da84a5c8f0e0c1d7ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:44.935370 14011 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6874fb018bc844b3aeeab7db8fc53e9a consensus state has no leader
I20251211 08:57:44.938858 14010 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_27e68705f39e45fda963e581c94a0d45 [id=26c0a9d9b1f84372b18a54f045eb768c] alter complete (version 3)
I20251211 08:57:44.955075 14418 tablet_service.cc:1505] Processing CreateTablet for tablet c6d419632b2a49e1975d4baa8ac85322 (DEFAULT_TABLE table=default.Table_9294d4662790426dbb71410c2c874bc0 [id=eec20447308f431092b2b3736ae03d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.955364 15506 raft_consensus.cc:493] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:44.955464 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 9821822304c948f3bfcf5d52901371a4 (DEFAULT_TABLE table=default.Table_4e15a6b7343e46c491019358d60e3589 [id=f53f94afd6924a859e896f3974c4c5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:44.955734 15506 raft_consensus.cc:515] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:44.957414 15506 leader_election.cc:290] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:44.967669 15512 consensus_queue.cc:1048] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:44.969465 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e813a4c73334c839bfe6af2119e259c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:44.992913 14905 ts_tablet_manager.cc:1929] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:44.993320 14905 log.cc:1199] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/174fefd70bcc41418a77a5716f97cd5f
I20251211 08:57:44.994057 14905 ts_tablet_manager.cc:1950] T 174fefd70bcc41418a77a5716f97cd5f P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:45.004150 14714 ts_tablet_manager.cc:1434] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.304s user 0.004s sys 0.002s
I20251211 08:57:45.008186 14421 tablet_service.cc:1505] Processing CreateTablet for tablet 02c7fbd61b2e42efaa0ae06ed1b1091c (DEFAULT_TABLE table=default.Table_9e37e9a3471a486692ffe27b9841b13a [id=ade2f567017f4807ad913bb1d62948b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:45.009590 14423 tablet_service.cc:1505] Processing CreateTablet for tablet 01310095bca4479da24185576091b748 (DEFAULT_TABLE table=default.Table_9ec80b196de446bf8a12c0ae014f783e [id=ee66bcc72bcf4627b996467b0825b1c0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:45.009857 14231 consensus_peers.cc:597] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ad600554c0946e0bcb0da6a42be2ffd. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:45.010531 14231 leader_election.cc:343] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: e961c5a3b5f640cbaea78b6185048d04
I20251211 08:57:45.010910 14231 leader_election.cc:304] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:45.011349 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01310095bca4479da24185576091b748. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:45.012140 14419 tablet_service.cc:1552] Processing DeleteTablet for tablet 4631dd0bc0c144e69a31de80de39a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:44 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:45.012900 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:39 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:45.006582 15299 tablet_replica.cc:333] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:45.015064 15299 raft_consensus.cc:2243] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:45.022341 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c7fbd61b2e42efaa0ae06ed1b1091c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:45.025924 14714 tablet_bootstrap.cc:492] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:45.035328 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 281) took 2699 ms. Trace:
I20251211 08:57:45.053766 14905 rpcz_store.cc:276] 1211 08:57:42.336043 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.336159 (+ 116us) service_pool.cc:225] Handling call
1211 08:57:44.402908 (+2066749us) ts_tablet_manager.cc:1151] Deleting tablet 174fefd70bcc41418a77a5716f97cd5f
1211 08:57:44.402967 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.505961 (+102994us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:44.992646 (+486685us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:45.035259 (+ 42613us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39081,"tablet-delete.queue_time_us":2066267}
I20251211 08:57:45.055056 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 174fefd70bcc41418a77a5716f97cd5f (table default.Table_fcf104ee591d470d806156189c2cac52 [id=b734e827456d451b9fe6f1dbf47fea1d]) successfully deleted
I20251211 08:57:45.055907 14905 tablet_replica.cc:333] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:45.056736 14905 raft_consensus.cc:2243] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:45.057485 14905 raft_consensus.cc:2272] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:45.059128 15299 raft_consensus.cc:2272] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:45.060776 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d419632b2a49e1975d4baa8ac85322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:45.062825 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9821822304c948f3bfcf5d52901371a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:45.092532 14905 ts_tablet_manager.cc:1916] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:45.131130 15299 ts_tablet_manager.cc:1916] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:45.134006 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 326838us
I20251211 08:57:45.135665 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c39cc816f4c438391bb4826ea1a508f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:45.141460 15030 tablet_bootstrap.cc:492] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:45.141855 15030 ts_tablet_manager.cc:1403] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.324s user 0.010s sys 0.031s
I20251211 08:57:45.150744 15030 raft_consensus.cc:359] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.151487 15030 raft_consensus.cc:385] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.151726 15030 raft_consensus.cc:740] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.164286 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f0e893169e89418ab044ae00cdb78176" } modify_external_catalogs: true
I20251211 08:57:45.164827 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f0e893169e89418ab044ae00cdb78176" } modify_external_catalogs: true
I20251211 08:57:45.165861 15030 consensus_queue.cc:260] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.166668 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 01310095bca4479da24185576091b748 (DEFAULT_TABLE table=default.Table_9ec80b196de446bf8a12c0ae014f783e [id=ee66bcc72bcf4627b996467b0825b1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:45.167649 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01310095bca4479da24185576091b748. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:45.167834 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 01310095bca4479da24185576091b748
I20251211 08:57:45.171676 14010 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_5bb3e684c5f445188aa010ca4b32db87" } new_table_name: "default.Table_e46c6e53d82c408980e7e825ef7a1084" modify_external_catalogs: true
I20251211 08:57:45.188210 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 01310095bca4479da24185576091b748 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:44 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:45.189019 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b39d68e7f1b84e4b88959642577ec864"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:45.192250 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 4631dd0bc0c144e69a31de80de39a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:44 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:45.204051 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b39d68e7f1b84e4b88959642577ec864 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:45.191999 14009 catalog_manager.cc:5654] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:45.206805 14009 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 90c8960c5bfa4a08905bd211654e4798 (table default.Table_d2367e8cf559401c8af423ce7425f5a8 [id=9f3e442a656a4a8080fbf847cb136b00]). Expected version 1 got 0
I20251211 08:57:45.207547 15030 ts_tablet_manager.cc:1434] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.065s user 0.006s sys 0.001s
I20251211 08:57:45.211907 15030 tablet_bootstrap.cc:492] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:45.211284 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c39cc816f4c438391bb4826ea1a508f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:45.215431 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 01310095bca4479da24185576091b748 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:44 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:45.217046 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 0d9b0ac51ef14f46ab477fe61b0e537e (DEFAULT_TABLE table=default.Table_98031d7c370b4bfea07903c599528db5 [id=6a36244d681c4ce8a33b88287c5b7b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:45.217438 14265 tablet_service.cc:1505] Processing CreateTablet for tablet 976d872841c8452190bfe3a50eb30cd0 (DEFAULT_TABLE table=default.Table_bbfc3f82c89f4bfab0a97bd49febabd4 [id=54eff67ea0a943598ef8c381396cfd1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:45.222934 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d9b0ac51ef14f46ab477fe61b0e537e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:45.223367 14904 ts_tablet_manager.cc:1929] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:45.224421 14904 log.cc:1199] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/174fefd70bcc41418a77a5716f97cd5f
I20251211 08:57:45.226269 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 976d872841c8452190bfe3a50eb30cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:45.227471 14904 ts_tablet_manager.cc:1950] T 174fefd70bcc41418a77a5716f97cd5f P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:45.233573 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 299) took 1521 ms. Trace:
I20251211 08:57:45.235173 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 0d9b0ac51ef14f46ab477fe61b0e537e (DEFAULT_TABLE table=default.Table_98031d7c370b4bfea07903c599528db5 [id=6a36244d681c4ce8a33b88287c5b7b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:45.236574 15030 tablet_bootstrap.cc:654] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.235635 14904 rpcz_store.cc:276] 1211 08:57:43.711977 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.712498 (+ 521us) service_pool.cc:225] Handling call
1211 08:57:44.762611 (+1050113us) ts_tablet_manager.cc:1151] Deleting tablet 174fefd70bcc41418a77a5716f97cd5f
1211 08:57:44.762662 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:44.845210 (+ 82548us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:45.222587 (+377377us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:45.233125 (+ 10538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049614}
W20251211 08:57:45.241587 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3cf2b4d14ba45d79e52b6fdb0497d58 consensus state has no leader
I20251211 08:57:45.242225 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 174fefd70bcc41418a77a5716f97cd5f (table default.Table_fcf104ee591d470d806156189c2cac52 [id=b734e827456d451b9fe6f1dbf47fea1d]) successfully deleted
I20251211 08:57:45.247215 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 300) took 1515 ms. Trace:
W20251211 08:57:45.258044 14544 consensus_peers.cc:597] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:45.259141 14382 leader_election.cc:343] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:45.262053 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet 01310095bca4479da24185576091b748 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:44 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:45.262444 15299 ts_tablet_manager.cc:1929] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
W20251211 08:57:45.263275 14385 leader_election.cc:343] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:45.263696 14385 leader_election.cc:304] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:45.265122 14904 rpcz_store.cc:276] 1211 08:57:43.731099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.731204 (+ 105us) service_pool.cc:225] Handling call
1211 08:57:45.244355 (+1513151us) ts_tablet_manager.cc:1151] Deleting tablet 5f7775994f684326b6312c802f6b084d
1211 08:57:45.244657 (+ 302us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.246589 (+ 1932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1512457}
I20251211 08:57:45.268081 15506 raft_consensus.cc:2749] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:45.268213 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 301) took 1477 ms. Trace:
W20251211 08:57:45.268975 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 5f7775994f684326b6312c802f6b084d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f7775994f684326b6312c802f6b084d already in progress: creating tablet
I20251211 08:57:45.257172 14722 tablet_bootstrap.cc:492] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:45.269889 14722 ts_tablet_manager.cc:1403] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.647s user 0.013s sys 0.008s
I20251211 08:57:45.272210 14722 raft_consensus.cc:359] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.275427 15030 tablet_bootstrap.cc:492] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:45.275882 15030 ts_tablet_manager.cc:1403] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.064s user 0.010s sys 0.001s
I20251211 08:57:45.277062 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e813a4c73334c839bfe6af2119e259c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:45.277642 14449 raft_consensus.cc:2468] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:45.278576 15030 raft_consensus.cc:359] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.279165 15030 raft_consensus.cc:385] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.279387 15030 raft_consensus.cc:740] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.279948 14722 raft_consensus.cc:385] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.280297 14722 raft_consensus.cc:740] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.268499 14904 rpcz_store.cc:276] 1211 08:57:43.790639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.790869 (+ 230us) service_pool.cc:225] Handling call
1211 08:57:45.267804 (+1476935us) ts_tablet_manager.cc:1151] Deleting tablet 74c8dd2b036c42a0bec79d3637f00ef4
1211 08:57:45.267858 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.268162 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1432760}
I20251211 08:57:45.281467 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 302) took 1466 ms. Trace:
W20251211 08:57:45.282171 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 74c8dd2b036c42a0bec79d3637f00ef4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74c8dd2b036c42a0bec79d3637f00ef4 already in progress: creating tablet
I20251211 08:57:45.282560 14542 leader_election.cc:304] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:45.282999 14904 rpcz_store.cc:276] 1211 08:57:43.815386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.815526 (+ 140us) service_pool.cc:225] Handling call
1211 08:57:45.281094 (+1465568us) ts_tablet_manager.cc:1151] Deleting tablet c112dfcc301648cda6f0f038b1f2af83
1211 08:57:45.281147 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.281411 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1443511}
W20251211 08:57:45.283895 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet c112dfcc301648cda6f0f038b1f2af83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c112dfcc301648cda6f0f038b1f2af83 already in progress: creating tablet
I20251211 08:57:45.284744 14904 tablet_replica.cc:333] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:45.284659 15515 raft_consensus.cc:2804] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:45.285027 15515 raft_consensus.cc:493] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:45.285323 14904 raft_consensus.cc:2243] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:45.285383 15515 raft_consensus.cc:3060] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:45.286120 14904 raft_consensus.cc:2272] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:45.287434 15540 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:45.286792 14722 consensus_queue.cc:260] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.290174 15514 consensus_queue.cc:1048] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:45.291059 14722 ts_tablet_manager.cc:1434] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20251211 08:57:45.291795 14722 tablet_bootstrap.cc:492] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:45.292549 15030 consensus_queue.cc:260] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.294512 14576 tablet_service.cc:1505] Processing CreateTablet for tablet 976d872841c8452190bfe3a50eb30cd0 (DEFAULT_TABLE table=default.Table_bbfc3f82c89f4bfab0a97bd49febabd4 [id=54eff67ea0a943598ef8c381396cfd1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:45.305292 15030 ts_tablet_manager.cc:1434] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.029s user 0.004s sys 0.000s
W20251211 08:57:45.306088 14010 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2ee6cab1d1742e08323842ad9c69b81 consensus state has no leader
I20251211 08:57:45.306339 15030 tablet_bootstrap.cc:492] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:45.308145 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 3838fd220d034b3595f351b906188339
I20251211 08:57:45.310743 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 3838fd220d034b3595f351b906188339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:45.312215 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:39 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:45.314636 14714 tablet_bootstrap.cc:654] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.325872 15030 tablet_bootstrap.cc:654] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.325891 15515 raft_consensus.cc:515] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.329717 15515 leader_election.cc:290] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:45.316905 15547 raft_consensus.cc:493] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:45.339815 15547 raft_consensus.cc:515] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.341501 15547 leader_election.cc:290] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:45.346169 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet 3838fd220d034b3595f351b906188339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:45.347272 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e813a4c73334c839bfe6af2119e259c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:45.347805 14444 raft_consensus.cc:3060] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:45.347747 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_5de14693e76648da88962b5448218b76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:45.349191 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5de14693e76648da88962b5448218b76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:45.353080 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_55467d689ec140f2a3ed64cc4ec0d806"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:45.355057 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55467d689ec140f2a3ed64cc4ec0d806 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:45.360741 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c2fb2d9564c43ee84b8dfcea8ec54b0" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:45.361323 14607 raft_consensus.cc:2468] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:45.366425 14904 ts_tablet_manager.cc:1916] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:45.368599 14382 leader_election.cc:304] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:45.358633 14444 raft_consensus.cc:2468] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:45.370146 14542 leader_election.cc:304] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:45.370997 15515 raft_consensus.cc:2804] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:45.371502 15515 raft_consensus.cc:697] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:45.372234 15515 consensus_queue.cc:237] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.365892 15512 raft_consensus.cc:2749] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:45.382434 15547 raft_consensus.cc:2804] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:45.382731 15547 raft_consensus.cc:493] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:45.383046 15547 raft_consensus.cc:3060] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:45.384047 14722 tablet_bootstrap.cc:654] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.383932 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e3e740f7ffee4b1683b5888496b1a384" } modify_external_catalogs: true
I20251211 08:57:45.389938 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e813a4c73334c839bfe6af2119e259c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:45.405143 14544 leader_election.cc:343] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:45.408646 13999 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 -> Peer 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015): Couldn't send request to peer 73702d54989f4959acfc4a6fdcede6ca. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20251211 08:57:45.420451 14722 tablet_bootstrap.cc:492] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:45.420855 14722 ts_tablet_manager.cc:1403] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.000s
I20251211 08:57:45.423036 14722 raft_consensus.cc:359] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.424016 14722 raft_consensus.cc:385] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.424906 14722 raft_consensus.cc:740] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.427116 14722 consensus_queue.cc:260] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.433337 14722 ts_tablet_manager.cc:1434] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20251211 08:57:45.435160 15030 tablet_bootstrap.cc:492] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:45.435679 15030 ts_tablet_manager.cc:1403] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.130s user 0.022s sys 0.010s
I20251211 08:57:45.437116 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d9b0ac51ef14f46ab477fe61b0e537e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:45.437958 15030 raft_consensus.cc:359] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.438563 15030 raft_consensus.cc:385] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.438841 15030 raft_consensus.cc:740] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.439216 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 976d872841c8452190bfe3a50eb30cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:45.439988 14722 tablet_bootstrap.cc:492] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:45.441587 15030 consensus_queue.cc:260] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.445168 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e3e740f7ffee4b1683b5888496b1a384" } modify_external_catalogs: true
I20251211 08:57:45.445050 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4f6449beeb5d4b80ac26c12176d3d209" } new_table_name: "default.Table_49a263a32c9940a5b61d8a430ebb61a7" modify_external_catalogs: true
I20251211 08:57:45.453482 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2ff00d533f5a4fb58be17cfd01686207"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:45.454936 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ff00d533f5a4fb58be17cfd01686207 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:45.455294 15030 ts_tablet_manager.cc:1434] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.019s user 0.002s sys 0.003s
I20251211 08:57:45.458297 14904 ts_tablet_manager.cc:1929] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:45.459038 15030 tablet_bootstrap.cc:492] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:45.459376 14904 log.cc:1199] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/8ad600554c0946e0bcb0da6a42be2ffd
I20251211 08:57:45.471673 14905 ts_tablet_manager.cc:1929] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:45.472074 14905 log.cc:1199] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/e961c5a3b5f640cbaea78b6185048d04
I20251211 08:57:45.472980 14905 ts_tablet_manager.cc:1950] T e961c5a3b5f640cbaea78b6185048d04 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:45.469579 14722 tablet_bootstrap.cc:654] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.478626 14904 ts_tablet_manager.cc:1950] T 8ad600554c0946e0bcb0da6a42be2ffd P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:45.479880 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 303) took 1478 ms. Trace:
I20251211 08:57:45.481936 15030 tablet_bootstrap.cc:654] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.485953 14904 rpcz_store.cc:276] 1211 08:57:44.001678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.001779 (+ 101us) service_pool.cc:225] Handling call
1211 08:57:45.284608 (+1282829us) ts_tablet_manager.cc:1151] Deleting tablet 8ad600554c0946e0bcb0da6a42be2ffd
1211 08:57:45.284651 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.379719 (+ 95068us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:45.453095 (+ 73376us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:45.479829 (+ 26734us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1282406}
I20251211 08:57:45.494159 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 8ad600554c0946e0bcb0da6a42be2ffd (table default.Table_1eb40c19ca894f47ba26da25462c5c3a [id=62e4ec0c216b41eebee5490eaba6ef5f]) successfully deleted
I20251211 08:57:45.497913 15030 tablet_bootstrap.cc:492] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:45.499244 15299 log.cc:1199] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/593df63696a04f85b00c301d93748e80
I20251211 08:57:45.499958 14722 tablet_bootstrap.cc:492] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:45.500165 15299 ts_tablet_manager.cc:1950] T 593df63696a04f85b00c301d93748e80 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:45.500329 14722 ts_tablet_manager.cc:1403] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.006s
I20251211 08:57:45.502615 14722 raft_consensus.cc:359] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.507412 14722 raft_consensus.cc:385] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.507668 14722 raft_consensus.cc:740] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.498266 15030 ts_tablet_manager.cc:1403] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.000s
I20251211 08:57:45.507894 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 306) took 1273 ms. Trace:
I20251211 08:57:45.509493 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_0442aa229734421dab4cf2f058c9c5e0" } new_table_name: "default.Table_78e567c0deac4f8d99c850bc335bbaf5" modify_external_catalogs: true
I20251211 08:57:45.511729 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8f9a86bf44d14c63af54ebaca1cfbb67" } modify_external_catalogs: true
I20251211 08:57:45.512195 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8f9a86bf44d14c63af54ebaca1cfbb67" } modify_external_catalogs: true
I20251211 08:57:45.508222 14722 consensus_queue.cc:260] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.509999 14904 rpcz_store.cc:276] 1211 08:57:44.234976 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.235110 (+ 134us) service_pool.cc:225] Handling call
1211 08:57:45.495047 (+1259937us) ts_tablet_manager.cc:1151] Deleting tablet 4c95565ce3954cc598d7cbd45321d7c9
1211 08:57:45.495097 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.507846 (+ 12749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1259025}
W20251211 08:57:45.520622 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 4c95565ce3954cc598d7cbd45321d7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c95565ce3954cc598d7cbd45321d7c9 already in progress: creating tablet
I20251211 08:57:45.520753 15030 raft_consensus.cc:359] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.521301 15030 raft_consensus.cc:385] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.521544 15030 raft_consensus.cc:740] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.521981 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 307) took 1286 ms. Trace:
W20251211 08:57:45.526489 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 263343us
I20251211 08:57:45.528169 14421 tablet_service.cc:1505] Processing CreateTablet for tablet 976d872841c8452190bfe3a50eb30cd0 (DEFAULT_TABLE table=default.Table_bbfc3f82c89f4bfab0a97bd49febabd4 [id=54eff67ea0a943598ef8c381396cfd1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:45.529253 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 976d872841c8452190bfe3a50eb30cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:45.530067 15030 consensus_queue.cc:260] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.533864 14904 rpcz_store.cc:276] 1211 08:57:44.235713 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.235829 (+ 116us) service_pool.cc:225] Handling call
1211 08:57:45.521540 (+1285711us) ts_tablet_manager.cc:1151] Deleting tablet ed0e90efe651499c929b1fb992712b96
1211 08:57:45.521599 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.521915 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1285206}
W20251211 08:57:45.534816 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet ed0e90efe651499c929b1fb992712b96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed0e90efe651499c929b1fb992712b96 already in progress: creating tablet
I20251211 08:57:45.535223 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 308) took 1246 ms. Trace:
I20251211 08:57:45.535698 14904 rpcz_store.cc:276] 1211 08:57:44.288948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.289442 (+ 494us) service_pool.cc:225] Handling call
1211 08:57:45.534904 (+1245462us) ts_tablet_manager.cc:1151] Deleting tablet 07f983604bfa474f9c9ce0c982d738fc
1211 08:57:45.534952 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.535177 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245002}
I20251211 08:57:45.529538 14722 ts_tablet_manager.cc:1434] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.029s user 0.006s sys 0.001s
I20251211 08:57:45.536597 15030 ts_tablet_manager.cc:1434] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
W20251211 08:57:45.536902 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 07f983604bfa474f9c9ce0c982d738fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07f983604bfa474f9c9ce0c982d738fc already in progress: creating tablet
I20251211 08:57:45.537442 14722 tablet_bootstrap.cc:492] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:45.543102 14722 tablet_bootstrap.cc:654] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.543668 15559 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:45.548437 14714 tablet_bootstrap.cc:492] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:45.548905 14714 ts_tablet_manager.cc:1403] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.524s user 0.010s sys 0.005s
I20251211 08:57:45.551133 14714 raft_consensus.cc:359] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:45.552412 13997 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_4f6449beeb5d4b80ac26c12176d3d209 [id=69dd3ef114034ac7a3d60b8a0fde70b6], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet e15483f539f24f1ba1b442663d66fd45 consensus state has no leader
I20251211 08:57:45.542937 14006 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 578404732eef4e51b66b6aebffa87cdb (table default.Table_27e68705f39e45fda963e581c94a0d45 [id=26c0a9d9b1f84372b18a54f045eb768c]). Expected version 3 got 2
W20251211 08:57:45.565709 14013 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e15483f539f24f1ba1b442663d66fd45 consensus state has no leader
I20251211 08:57:45.567474 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 7b90b0c944174b2a82696e3fee806934
I20251211 08:57:45.570533 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b90b0c944174b2a82696e3fee806934 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:45.571770 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b90b0c944174b2a82696e3fee806934 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:45.575109 14904 tablet_replica.cc:333] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:45.575811 14904 raft_consensus.cc:2243] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:45.576290 14904 raft_consensus.cc:2272] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:45.578852 14904 ts_tablet_manager.cc:1916] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:45.581979 15030 tablet_bootstrap.cc:492] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:45.583845 14607 raft_consensus.cc:1275] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Refusing update from remote peer b276f77f87254a5d909c5ab940a39256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:57:45.597389 14257 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:46838 (request call id 53) took 1055 ms. Trace:
I20251211 08:57:45.597720 14257 rpcz_store.cc:276] 1211 08:57:44.542145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.545964 (+ 3819us) service_pool.cc:225] Handling call
1211 08:57:45.524178 (+978214us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b5400161054. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f141cc5a1 00007f0f1425cb78 00007f0f1529d20d 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:57:45.597169 (+ 72991us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":243412736}
I20251211 08:57:45.598436 15030 tablet_bootstrap.cc:654] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.603253 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 282) took 3266 ms. Trace:
I20251211 08:57:45.614102 14008 master_service.cc:954] ReplaceTablet: received request to replace tablet 7af7c9ab27a64a1681f9874a922e986f from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:45.614553 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c2fb2d9564c43ee84b8dfcea8ec54b0" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:45.618582 14722 tablet_bootstrap.cc:492] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:45.618999 14722 ts_tablet_manager.cc:1403] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.001s
I20251211 08:57:45.619653 15030 tablet_bootstrap.cc:492] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:45.620046 15030 ts_tablet_manager.cc:1403] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.000s
I20251211 08:57:45.621022 14722 raft_consensus.cc:359] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.621546 14722 raft_consensus.cc:385] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.621762 14722 raft_consensus.cc:740] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.624994 15554 raft_consensus.cc:493] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:45.625380 15554 raft_consensus.cc:515] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.623801 15030 raft_consensus.cc:359] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.626175 15030 raft_consensus.cc:385] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.626422 15030 raft_consensus.cc:740] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.622331 14722 consensus_queue.cc:260] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.628157 15554 leader_election.cc:290] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:45.632288 14722 ts_tablet_manager.cc:1434] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.013s user 0.002s sys 0.002s
I20251211 08:57:45.603638 14905 rpcz_store.cc:276] 1211 08:57:42.337167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.337701 (+ 534us) service_pool.cc:225] Handling call
1211 08:57:45.055747 (+2718046us) ts_tablet_manager.cc:1151] Deleting tablet e961c5a3b5f640cbaea78b6185048d04
1211 08:57:45.055793 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.148338 (+ 92545us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:45.471403 (+323065us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:45.603176 (+131773us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2717595}
I20251211 08:57:45.633944 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet e961c5a3b5f640cbaea78b6185048d04 (table default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]) successfully deleted
W20251211 08:57:45.635996 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4dcd09f7c6364d74bb33503b5bbfea96 consensus state has no leader
I20251211 08:57:45.630035 15030 consensus_queue.cc:260] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.637328 15515 raft_consensus.cc:493] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:45.637809 15515 raft_consensus.cc:515] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.639587 15515 leader_election.cc:290] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:45.642899 15030 ts_tablet_manager.cc:1434] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.023s user 0.005s sys 0.002s
I20251211 08:57:45.647006 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f938359354b545cfbac560b7d7888b9f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:45.648276 14544 leader_election.cc:343] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:45.652693 14231 leader_election.cc:343] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: dca342fd9de241b5884600199f06fbeb
I20251211 08:57:45.658420 14904 ts_tablet_manager.cc:1929] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:45.661586 14714 raft_consensus.cc:385] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.661607 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dca342fd9de241b5884600199f06fbeb" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:45.661856 14714 raft_consensus.cc:740] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.553563 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 334) took 2441 ms. Trace:
I20251211 08:57:45.663131 14230 leader_election.cc:304] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:45.662950 15299 rpcz_store.cc:276] 1211 08:57:43.111949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.112052 (+ 103us) service_pool.cc:225] Handling call
1211 08:57:44.790853 (+1678801us) ts_tablet_manager.cc:1151] Deleting tablet 593df63696a04f85b00c301d93748e80
1211 08:57:44.790883 (+ 30us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.255603 (+464720us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:45.262269 (+ 6666us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:45.553513 (+291244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1678293}
I20251211 08:57:45.663686 14904 log.cc:1199] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/e961c5a3b5f640cbaea78b6185048d04
I20251211 08:57:45.664059 15554 raft_consensus.cc:2749] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:45.664130 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 337) took 2262 ms. Trace:
I20251211 08:57:45.664449 14904 ts_tablet_manager.cc:1950] T e961c5a3b5f640cbaea78b6185048d04 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:45.664419 15299 rpcz_store.cc:276] 1211 08:57:43.401838 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.410920 (+ 9082us) service_pool.cc:225] Handling call
1211 08:57:45.663795 (+2252875us) ts_tablet_manager.cc:1151] Deleting tablet dd14de369e084828afb8a924c46c649b
1211 08:57:45.663832 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.664078 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2235316}
I20251211 08:57:45.665267 15515 raft_consensus.cc:493] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:45.665632 15515 raft_consensus.cc:515] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.667037 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 310) took 1314 ms. Trace:
I20251211 08:57:45.667259 15515 leader_election.cc:290] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:45.667312 14904 rpcz_store.cc:276] 1211 08:57:44.352306 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.352414 (+ 108us) service_pool.cc:225] Handling call
1211 08:57:45.574966 (+1222552us) ts_tablet_manager.cc:1151] Deleting tablet e961c5a3b5f640cbaea78b6185048d04
1211 08:57:45.575015 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.594749 (+ 19734us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:45.658210 (+ 63461us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:45.666990 (+ 8780us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222142}
I20251211 08:57:45.668291 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet e961c5a3b5f640cbaea78b6185048d04 (table default.Table_4c07a2c494ed47d29a486da1035027e3 [id=106fceb173604910b123dae0d61809dd]) successfully deleted
I20251211 08:57:45.668437 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cbd18e927c843e7b0aee7e0ccb1b06d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:45.669304 14542 leader_election.cc:343] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:45.669579 14722 tablet_bootstrap.cc:492] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:45.557521 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 0d9b0ac51ef14f46ab477fe61b0e537e (DEFAULT_TABLE table=default.Table_98031d7c370b4bfea07903c599528db5 [id=6a36244d681c4ce8a33b88287c5b7b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:45.576893 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b90b0c944174b2a82696e3fee806934 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:45.585125 15486 consensus_queue.cc:1048] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:45.672114 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d9b0ac51ef14f46ab477fe61b0e537e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:45.676656 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cbd18e927c843e7b0aee7e0ccb1b06d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:45.677098 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 312) took 1133 ms. Trace:
I20251211 08:57:45.677410 14904 rpcz_store.cc:276] 1211 08:57:44.543514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.550963 (+ 7449us) service_pool.cc:225] Handling call
1211 08:57:45.676805 (+1125842us) ts_tablet_manager.cc:1151] Deleting tablet 5d8d84e594ea47f3bc9127cc587e8aa7
1211 08:57:45.676844 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.677050 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124940}
W20251211 08:57:45.677703 14544 leader_election.cc:343] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:45.678030 14544 leader_election.cc:304] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
W20251211 08:57:45.678539 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d8d84e594ea47f3bc9127cc587e8aa7 already in progress: creating tablet
I20251211 08:57:45.678843 15515 raft_consensus.cc:2749] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:45.681744 15554 raft_consensus.cc:493] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:45.615209 15547 raft_consensus.cc:515] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.645408 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f938359354b545cfbac560b7d7888b9f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:45.682139 15554 raft_consensus.cc:515] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:45.683244 14542 leader_election.cc:343] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:45.683585 14542 leader_election.cc:304] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:45.683861 15554 leader_election.cc:290] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:45.683967 14722 tablet_bootstrap.cc:654] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.685343 15515 raft_consensus.cc:2749] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:45.585901 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 3838fd220d034b3595f351b906188339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:45.685954 14231 leader_election.cc:343] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: c53035e83710403bac3a2b401c8c97ad
I20251211 08:57:45.687450 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c53035e83710403bac3a2b401c8c97ad" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:45.688836 14230 leader_election.cc:304] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
W20251211 08:57:45.689612 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 4631dd0bc0c144e69a31de80de39a76a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4631dd0bc0c144e69a31de80de39a76a already in progress: creating tablet
I20251211 08:57:45.689967 15512 raft_consensus.cc:2749] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:45.691404 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 01310095bca4479da24185576091b748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01310095bca4479da24185576091b748 already in progress: creating tablet
I20251211 08:57:45.691808 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 342) took 1961 ms. Trace:
W20251211 08:57:45.691928 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 3838fd220d034b3595f351b906188339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3838fd220d034b3595f351b906188339 already in progress: creating tablet
I20251211 08:57:45.691179 14904 tablet_replica.cc:333] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:45.693897 14904 raft_consensus.cc:2243] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:45.694291 14904 raft_consensus.cc:2272] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:45.696192 14904 ts_tablet_manager.cc:1916] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:45.697460 15515 raft_consensus.cc:493] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:45.697849 15515 raft_consensus.cc:515] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.700588 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2016208a33dc4a5cadd7ced8e85ccc65" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:45.701256 14299 raft_consensus.cc:2468] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:45.702142 14544 leader_election.cc:304] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:45.702821 15514 raft_consensus.cc:2804] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20251211 08:57:45.702956 14542 leader_election.cc:343] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 2016208a33dc4a5cadd7ced8e85ccc65
I20251211 08:57:45.703125 15514 raft_consensus.cc:493] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:45.703476 15514 raft_consensus.cc:3060] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:57:45.703763 15515 leader_election.cc:290] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:45.706027 15515 raft_consensus.cc:493] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:45.706461 15515 raft_consensus.cc:515] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.709101 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f32a1ccedd4f15bf8c79bc3f16e6bb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:45.710222 14542 leader_election.cc:343] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:45.711369 15514 raft_consensus.cc:515] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.712366 15515 leader_election.cc:290] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:45.716511 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2016208a33dc4a5cadd7ced8e85ccc65" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:45.717092 14298 raft_consensus.cc:3060] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 2
I20251211 08:57:45.718694 15580 tablet.cc:1722] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:45.723167 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f32a1ccedd4f15bf8c79bc3f16e6bb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:45.724362 14544 leader_election.cc:343] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:45.724792 14544 leader_election.cc:304] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:45.725359 14722 tablet_bootstrap.cc:492] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:45.726112 14722 ts_tablet_manager.cc:1403] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.057s user 0.015s sys 0.001s
I20251211 08:57:45.726063 14298 raft_consensus.cc:2468] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 2.
I20251211 08:57:45.727275 14544 leader_election.cc:304] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:45.727910 15440 raft_consensus.cc:2749] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:45.728363 15440 raft_consensus.cc:2804] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader election won for term 2
I20251211 08:57:45.728729 15440 raft_consensus.cc:697] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:45.729437 15440 consensus_queue.cc:237] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.732770 14722 raft_consensus.cc:359] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.733304 14722 raft_consensus.cc:385] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.733556 14722 raft_consensus.cc:740] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.734138 14722 consensus_queue.cc:260] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.736449 14722 ts_tablet_manager.cc:1434] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20251211 08:57:45.737252 14722 tablet_bootstrap.cc:492] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:45.741498 15581 tablet.cc:1722] T 90c8960c5bfa4a08905bd211654e4798 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:45.744601 15547 leader_election.cc:290] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:45.754520 14385 leader_election.cc:343] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:45.692075 15299 rpcz_store.cc:276] 1211 08:57:43.730267 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.740086 (+ 9819us) service_pool.cc:225] Handling call
1211 08:57:45.691512 (+1951426us) ts_tablet_manager.cc:1151] Deleting tablet 5f7775994f684326b6312c802f6b084d
1211 08:57:45.691554 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.691758 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1948851}
I20251211 08:57:45.758075 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 343) took 1968 ms. Trace:
I20251211 08:57:45.662376 14714 consensus_queue.cc:260] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.760880 14007 master_service.cc:954] ReplaceTablet: received request to replace tablet 6874fb018bc844b3aeeab7db8fc53e9a from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:45.747064 15030 tablet_bootstrap.cc:492] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:45.748456 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c2fb2d9564c43ee84b8dfcea8ec54b0" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:45.754933 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 279) took 3421 ms. Trace:
I20251211 08:57:45.764600 14905 rpcz_store.cc:276] 1211 08:57:42.334037 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.374921 (+ 40884us) service_pool.cc:225] Handling call
1211 08:57:45.637604 (+3262683us) ts_tablet_manager.cc:1151] Deleting tablet 42720f5f5bf445f2b72e513ad4a6d95f
1211 08:57:45.691405 (+ 53801us) spinlock_profiling.cc:243] Waited 42.2 ms on lock 0x7b5800000098. stack: 00007f0c13f80083 00007f0c13f7ff80 00007f0c13af5574 0000000000508744 00000000005086d9 00007f0c13df2781 00007f0c13f74ac1 00007f0c13f750f2 00007f0c1cdc2419 00007f0c1c018108 00007f0c1c0de9a9 00007f0c1c0defa3 00007f0c1c10ff80 00007f0c1c0f654d 00007f0c1c0f64f9 00007f0c1c0f6489
1211 08:57:45.754723 (+ 63318us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.754886 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":96961792,"tablet-delete.queue_time_us":3262035}
I20251211 08:57:45.765558 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_e4cd2059809846e78020a7513c96df7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:45.766158 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 42720f5f5bf445f2b72e513ad4a6d95f because the tablet was not found. No further retry: Not found: Tablet not found: 42720f5f5bf445f2b72e513ad4a6d95f
W20251211 08:57:45.767474 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4cd2059809846e78020a7513c96df7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:45.770421 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ca924cbdb1e6438aa8cf80713dad7885"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:45.772205 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca924cbdb1e6438aa8cf80713dad7885 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:45.774921 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_27e68705f39e45fda963e581c94a0d45" } new_table_name: "default.Table_bf148c361a2346e88e4c9e4430b1abca" modify_external_catalogs: true
I20251211 08:57:45.778491 14005 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 4dbf40285b4c4465b68fa5cf34dbe900
I20251211 08:57:45.790009 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_dcb6d015c5c543689fc117f18bdff260" } modify_external_catalogs: true
I20251211 08:57:45.790689 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_dcb6d015c5c543689fc117f18bdff260" } modify_external_catalogs: true
W20251211 08:57:45.790977 14385 consensus_peers.cc:597] T 90c8960c5bfa4a08905bd211654e4798 P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:45.791666 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 593df63696a04f85b00c301d93748e80 (table default.Table_da883b3cac244bc0aee81f7c17ac81a6 [id=ab93470ba8f54070be718c240d49aa13]) successfully deleted
I20251211 08:57:45.795337 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet c53035e83710403bac3a2b401c8c97ad with delete_type TABLET_DATA_DELETED (replaced by tablet dd14de369e084828afb8a924c46c649b) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:45.795403 14714 ts_tablet_manager.cc:1434] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.246s user 0.005s sys 0.000s
I20251211 08:57:45.797806 14722 tablet_bootstrap.cc:654] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.758414 15299 rpcz_store.cc:276] 1211 08:57:43.789220 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.790906 (+ 1686us) service_pool.cc:225] Handling call
1211 08:57:45.757766 (+1966860us) ts_tablet_manager.cc:1151] Deleting tablet 74c8dd2b036c42a0bec79d3637f00ef4
1211 08:57:45.757802 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.758029 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1966317}
W20251211 08:57:45.799517 13902 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 47923.3us
I20251211 08:57:45.802031 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 344) took 1980 ms. Trace:
I20251211 08:57:45.802616 15514 leader_election.cc:290] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:45.804538 14542 leader_election.cc:343] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 2016208a33dc4a5cadd7ced8e85ccc65
I20251211 08:57:45.805374 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 6874fb018bc844b3aeeab7db8fc53e9a
I20251211 08:57:45.806463 14007 catalog_manager.cc:6724] ReplaceTablet: tablet 6874fb018bc844b3aeeab7db8fc53e9a deleted and replaced by tablet 593a90d88b194bb5a1c37e0e7e9097ef
I20251211 08:57:45.807513 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 6874fb018bc844b3aeeab7db8fc53e9a with delete_type TABLET_DATA_DELETED (replaced by tablet 593a90d88b194bb5a1c37e0e7e9097ef) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:45.808072 14904 ts_tablet_manager.cc:1929] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:45.808444 14904 log.cc:1199] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/7b90b0c944174b2a82696e3fee806934
I20251211 08:57:45.809078 14904 ts_tablet_manager.cc:1950] T 7b90b0c944174b2a82696e3fee806934 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:57:45.809247 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet dd14de369e084828afb8a924c46c649b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd14de369e084828afb8a924c46c649b already in progress: creating tablet
I20251211 08:57:45.814963 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet e961c5a3b5f640cbaea78b6185048d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:33 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:45.815088 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet dca342fd9de241b5884600199f06fbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:45.820500 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 5f7775994f684326b6312c802f6b084d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f7775994f684326b6312c802f6b084d already in progress: creating tablet
I20251211 08:57:45.827260 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 2016208a33dc4a5cadd7ced8e85ccc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:35 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:45.827600 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 6874fb018bc844b3aeeab7db8fc53e9a with delete_type TABLET_DATA_DELETED (replaced by tablet 593a90d88b194bb5a1c37e0e7e9097ef) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:45.828477 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c2fb2d9564c43ee84b8dfcea8ec54b0" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:45.829310 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 7b90b0c944174b2a82696e3fee806934 (table default.Table_e3e740f7ffee4b1683b5888496b1a384 [id=0f8c030be5504d859ef25176b6cbb4b1]) successfully deleted
I20251211 08:57:45.830238 14904 tablet_replica.cc:333] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:45.833192 14904 raft_consensus.cc:2243] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:45.835314 14904 raft_consensus.cc:2272] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:45.842221 15554 raft_consensus.cc:493] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:45.842703 15554 raft_consensus.cc:515] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.844779 15554 leader_election.cc:290] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:45.847450 14231 leader_election.cc:343] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 9ccfcacb5b294a628926d34cd31b137e
I20251211 08:57:45.856194 14722 tablet_bootstrap.cc:492] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:45.856561 14722 ts_tablet_manager.cc:1403] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.119s user 0.011s sys 0.016s
I20251211 08:57:45.858609 14722 raft_consensus.cc:359] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.859139 14722 raft_consensus.cc:385] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.858884 15299 rpcz_store.cc:276] 1211 08:57:43.821491 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.823565 (+ 2074us) service_pool.cc:225] Handling call
1211 08:57:45.801722 (+1978157us) ts_tablet_manager.cc:1151] Deleting tablet c112dfcc301648cda6f0f038b1f2af83
1211 08:57:45.801761 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:45.801985 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1974555}
I20251211 08:57:45.859377 14722 raft_consensus.cc:740] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.859897 15299 tablet_replica.cc:333] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:45.859889 14722 consensus_queue.cc:260] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.861591 14714 tablet_bootstrap.cc:492] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:45.765812 14905 tablet_replica.cc:333] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:45.864094 14905 raft_consensus.cc:2243] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:45.869844 14722 ts_tablet_manager.cc:1434] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20251211 08:57:45.873453 14722 tablet_bootstrap.cc:492] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:45.873852 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 74c8dd2b036c42a0bec79d3637f00ef4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74c8dd2b036c42a0bec79d3637f00ef4 already in progress: creating tablet
I20251211 08:57:45.886894 14904 ts_tablet_manager.cc:1916] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:45.888139 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet c112dfcc301648cda6f0f038b1f2af83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c112dfcc301648cda6f0f038b1f2af83 already in progress: creating tablet
W20251211 08:57:45.889210 14385 leader_election.cc:343] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:45.890187 14722 tablet_bootstrap.cc:654] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.900731 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_41bf1acc3ec84195b9b1034fcdbe1e42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:45.897385 15299 raft_consensus.cc:2243] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:45.904927 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 6874fb018bc844b3aeeab7db8fc53e9a with delete_type TABLET_DATA_DELETED (replaced by tablet 593a90d88b194bb5a1c37e0e7e9097ef) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:45.905167 15554 raft_consensus.cc:493] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:45.903690 15299 raft_consensus.cc:2272] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:45.905563 15554 raft_consensus.cc:515] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.907456 15554 leader_election.cc:290] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:45.902455 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41bf1acc3ec84195b9b1034fcdbe1e42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:45.909750 14231 leader_election.cc:343] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: f909d82e0136470a810e1063d6769868
I20251211 08:57:45.910595 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_8855dc73d29241daa7d7c423bcae474b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:45.904567 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ea1935de13854b0b82e1382effe6cfbb" } new_table_name: "default.Table_cd9ec56dc08c48779e88a778d2917bb0" modify_external_catalogs: true
W20251211 08:57:45.915867 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8855dc73d29241daa7d7c423bcae474b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:45.926096 14714 tablet_bootstrap.cc:654] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.928613 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ccfcacb5b294a628926d34cd31b137e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:45.929898 14905 raft_consensus.cc:2272] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:45.865132 15515 raft_consensus.cc:493] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:45.929126 14607 raft_consensus.cc:2468] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:45.933153 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:45.935060 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ccfcacb5b294a628926d34cd31b137e with delete_type TABLET_DATA_DELETED (replaced by tablet 1065085c701f48c9ad7206c410ce5985) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:45.945521 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 54739us
I20251211 08:57:45.951004 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 578404732eef4e51b66b6aebffa87cdb (table default.Table_bf148c361a2346e88e4c9e4430b1abca [id=26c0a9d9b1f84372b18a54f045eb768c]). Expected version 4 got 3
I20251211 08:57:45.927153 14604 raft_consensus.cc:3060] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:45.954500 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet f909d82e0136470a810e1063d6769868 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:34 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:45.961179 14722 tablet_bootstrap.cc:492] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:45.961563 14722 ts_tablet_manager.cc:1403] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.090s user 0.020s sys 0.009s
I20251211 08:57:45.963315 15030 tablet_bootstrap.cc:654] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:45.963615 14722 raft_consensus.cc:359] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.964164 14722 raft_consensus.cc:385] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:45.964403 14722 raft_consensus.cc:740] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:45.967132 14722 consensus_queue.cc:260] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:45.972481 14904 ts_tablet_manager.cc:1929] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:45.972790 14904 log.cc:1199] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/c53035e83710403bac3a2b401c8c97ad
I20251211 08:57:45.973351 14904 ts_tablet_manager.cc:1950] T c53035e83710403bac3a2b401c8c97ad P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:45.974462 14722 ts_tablet_manager.cc:1434] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20251211 08:57:45.982846 14905 ts_tablet_manager.cc:1916] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:46.006459 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7558ae666eb04c7b96e3d1151f22fbe6" } modify_external_catalogs: true
I20251211 08:57:46.006989 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7558ae666eb04c7b96e3d1151f22fbe6" } modify_external_catalogs: true
I20251211 08:57:45.930656 15515 raft_consensus.cc:515] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.017906 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eec7694087145d8a49dd93904f95980 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:46.019762 14904 tablet_replica.cc:333] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:46.020450 14904 raft_consensus.cc:2243] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:46.020846 14904 raft_consensus.cc:2272] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:46.022379 14904 ts_tablet_manager.cc:1916] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:46.024505 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet c53035e83710403bac3a2b401c8c97ad (table default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]) successfully deleted
W20251211 08:57:46.025133 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet e961c5a3b5f640cbaea78b6185048d04 because the tablet was not found. No further retry: Not found: Tablet not found: e961c5a3b5f640cbaea78b6185048d04
I20251211 08:57:46.025357 14722 tablet_bootstrap.cc:492] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:46.030316 14722 tablet_bootstrap.cc:654] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:46.003553 15299 ts_tablet_manager.cc:1916] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:46.049830 14013 master_service.cc:954] ReplaceTablet: received request to replace tablet 14fc87cb76024b5b8227dfe9939e1c89 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:46.050328 14905 ts_tablet_manager.cc:1929] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20251211 08:57:46.056284 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ab48f6288494005ad42208e2e97ddf2 consensus state has no leader
I20251211 08:57:46.050659 14905 log.cc:1199] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/12379255548f49dfad17f3840a0c86bb
I20251211 08:57:46.072288 14905 ts_tablet_manager.cc:1950] T 12379255548f49dfad17f3840a0c86bb P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:46.091754 14230 leader_election.cc:304] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:46.092717 15554 raft_consensus.cc:2804] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:46.093047 15554 raft_consensus.cc:493] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:46.093358 15554 raft_consensus.cc:3060] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:46.099277 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f909d82e0136470a810e1063d6769868" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:46.100119 14605 raft_consensus.cc:2393] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:46.089623 15515 leader_election.cc:290] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:46.104171 14604 raft_consensus.cc:2468] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:46.106976 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f13b896c35754c0a94cc5b238778ecd2" } new_table_name: "default.Table_6cdb098ed94b446daec0ae76c262dbb8" modify_external_catalogs: true
I20251211 08:57:46.110283 15554 raft_consensus.cc:515] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:46.113406 14231 leader_election.cc:343] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 9ccfcacb5b294a628926d34cd31b137e
I20251211 08:57:46.114496 15554 leader_election.cc:290] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:46.125499 14298 raft_consensus.cc:3060] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:46.128165 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 283) took 3508 ms. Trace:
I20251211 08:57:46.128603 14905 rpcz_store.cc:276] 1211 08:57:42.620154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.620270 (+ 116us) service_pool.cc:225] Handling call
1211 08:57:45.765686 (+3145416us) ts_tablet_manager.cc:1151] Deleting tablet 12379255548f49dfad17f3840a0c86bb
1211 08:57:45.765719 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.014390 (+248671us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:46.050121 (+ 35731us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:46.128092 (+ 77971us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62947,"tablet-delete.queue_time_us":3144960}
I20251211 08:57:46.132386 14904 ts_tablet_manager.cc:1929] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:46.132781 14904 log.cc:1199] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/dca342fd9de241b5884600199f06fbeb
I20251211 08:57:46.133466 14904 ts_tablet_manager.cc:1950] T dca342fd9de241b5884600199f06fbeb P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:46.134652 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet dca342fd9de241b5884600199f06fbeb (table default.Table_f627382c623a4f818caf527706e776ab [id=22f3369559b34f5aa54c92c1bb01c2cb]) successfully deleted
I20251211 08:57:46.143513 14904 tablet_replica.cc:333] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:46.144196 14904 raft_consensus.cc:2243] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:46.144583 14904 raft_consensus.cc:2272] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:46.146202 14904 ts_tablet_manager.cc:1916] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:46.148694 14714 tablet_bootstrap.cc:492] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:46.149070 14714 ts_tablet_manager.cc:1403] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.334s user 0.007s sys 0.003s
I20251211 08:57:46.156265 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 14fc87cb76024b5b8227dfe9939e1c89
I20251211 08:57:46.158452 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 14fc87cb76024b5b8227dfe9939e1c89 with delete_type TABLET_DATA_DELETED (replaced by tablet df7611ed7bf5459e9fc81d172ed0f57d) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:46.160679 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45c7d841bd3643a799a8b96e82d6d675" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:46.162997 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 14fc87cb76024b5b8227dfe9939e1c89 with delete_type TABLET_DATA_DELETED (replaced by tablet df7611ed7bf5459e9fc81d172ed0f57d) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:46.165597 14905 tablet_replica.cc:333] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:46.167332 14013 catalog_manager.cc:6724] ReplaceTablet: tablet 14fc87cb76024b5b8227dfe9939e1c89 deleted and replaced by tablet df7611ed7bf5459e9fc81d172ed0f57d
I20251211 08:57:46.130272 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 12379255548f49dfad17f3840a0c86bb (table default.Table_da9fb81b38a14423b6191db9f7a7c3df [id=e10eaa167f0a46c39674a5e2212f2226]) successfully deleted
I20251211 08:57:46.169806 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 14fc87cb76024b5b8227dfe9939e1c89 with delete_type TABLET_DATA_DELETED (replaced by tablet df7611ed7bf5459e9fc81d172ed0f57d) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:46.173273 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45c7d841bd3643a799a8b96e82d6d675" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:46.174623 14722 tablet_bootstrap.cc:492] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:46.179350 15514 raft_consensus.cc:493] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:46.179986 15030 tablet_bootstrap.cc:492] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:46.179769 15514 raft_consensus.cc:515] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.180348 15030 ts_tablet_manager.cc:1403] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.433s user 0.016s sys 0.000s
I20251211 08:57:46.182104 15514 leader_election.cc:290] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:46.183537 14722 ts_tablet_manager.cc:1403] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.160s user 0.006s sys 0.020s
I20251211 08:57:46.190052 14722 raft_consensus.cc:359] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.190584 14722 raft_consensus.cc:385] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:46.191926 14722 raft_consensus.cc:740] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:46.199026 14722 consensus_queue.cc:260] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.201516 14722 ts_tablet_manager.cc:1434] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.018s user 0.001s sys 0.005s
I20251211 08:57:46.203651 14722 tablet_bootstrap.cc:492] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:46.207332 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17ab48721a5c48a7b8a5550be658837d consensus state has no leader
I20251211 08:57:46.215993 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6fd146e5066b4fb9b651b986d7992aa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:46.216567 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c39cc816f4c438391bb4826ea1a508f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:46.217159 14448 raft_consensus.cc:2468] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
W20251211 08:57:46.217497 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fd146e5066b4fb9b651b986d7992aa8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:46.222749 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7ebe2394d4924d5b9e1345f599ad502d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:46.223205 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_633182cd246844aba2af953c29c5d9e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:46.223822 14714 raft_consensus.cc:359] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:46.224306 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ebe2394d4924d5b9e1345f599ad502d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:46.224458 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_633182cd246844aba2af953c29c5d9e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:46.224524 14714 raft_consensus.cc:385] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:46.224766 14714 raft_consensus.cc:740] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:46.227705 15299 ts_tablet_manager.cc:1929] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:46.228042 15299 log.cc:1199] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/4e813a4c73334c839bfe6af2119e259c
I20251211 08:57:46.228643 15299 ts_tablet_manager.cc:1950] T 4e813a4c73334c839bfe6af2119e259c P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:46.228395 14714 consensus_queue.cc:260] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.230333 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 345) took 2368 ms. Trace:
I20251211 08:57:46.233021 14714 ts_tablet_manager.cc:1434] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.084s user 0.006s sys 0.001s
I20251211 08:57:46.237901 14904 ts_tablet_manager.cc:1929] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:46.238121 14714 tablet_bootstrap.cc:492] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:46.241135 14722 tablet_bootstrap.cc:654] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:46.238292 14904 log.cc:1199] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/2016208a33dc4a5cadd7ced8e85ccc65
I20251211 08:57:46.247310 14904 ts_tablet_manager.cc:1950] T 2016208a33dc4a5cadd7ced8e85ccc65 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:46.249696 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 2016208a33dc4a5cadd7ced8e85ccc65 (table default.Table_face70f737874a76a60d1f4ea105380c [id=f93e9bee7bdd443c9eb3d158f1e40d06]) successfully deleted
W20251211 08:57:46.252592 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 6874fb018bc844b3aeeab7db8fc53e9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6874fb018bc844b3aeeab7db8fc53e9a already in progress: creating tablet
I20251211 08:57:46.255151 14904 tablet_replica.cc:333] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:46.230620 15299 rpcz_store.cc:276] 1211 08:57:43.862226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.862347 (+ 121us) service_pool.cc:225] Handling call
1211 08:57:45.859741 (+1997394us) ts_tablet_manager.cc:1151] Deleting tablet 4e813a4c73334c839bfe6af2119e259c
1211 08:57:45.859786 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.151346 (+291560us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:46.227480 (+ 76134us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:46.230276 (+ 2796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1996922}
I20251211 08:57:46.260258 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 347) took 2030 ms. Trace:
I20251211 08:57:46.260365 14010 master_service.cc:954] ReplaceTablet: received request to replace tablet c6d419632b2a49e1975d4baa8ac85322 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:46.260546 15299 rpcz_store.cc:276] 1211 08:57:44.229441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.229565 (+ 124us) service_pool.cc:225] Handling call
1211 08:57:46.259966 (+2030401us) ts_tablet_manager.cc:1151] Deleting tablet 4c95565ce3954cc598d7cbd45321d7c9
1211 08:57:46.260002 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.260210 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2029859}
I20251211 08:57:46.268280 14722 tablet_bootstrap.cc:492] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:46.268687 14722 ts_tablet_manager.cc:1403] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.008s
I20251211 08:57:46.270807 14722 raft_consensus.cc:359] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.271397 14722 raft_consensus.cc:385] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:46.271607 14722 raft_consensus.cc:740] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:46.272199 14722 consensus_queue.cc:260] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.281082 14722 ts_tablet_manager.cc:1434] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20251211 08:57:46.281832 14722 tablet_bootstrap.cc:492] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:46.285058 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_d0cc7be666b34403b1626f0999117efe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:46.287463 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_20ad075bb5ab4a5dadc32bcfe94b0322" } new_table_name: "default.Table_10d2cf5a95b9442da2fb73a50e9cbd8d" modify_external_catalogs: true
I20251211 08:57:46.291494 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_49a263a32c9940a5b61d8a430ebb61a7 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 3 got 0
I20251211 08:57:46.291755 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 348) took 2058 ms. Trace:
I20251211 08:57:46.292045 15299 rpcz_store.cc:276] 1211 08:57:44.232996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.233134 (+ 138us) service_pool.cc:225] Handling call
1211 08:57:46.261207 (+2028073us) ts_tablet_manager.cc:1151] Deleting tablet ed0e90efe651499c929b1fb992712b96
1211 08:57:46.291432 (+ 30225us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.291698 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027618}
W20251211 08:57:46.286597 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0cc7be666b34403b1626f0999117efe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:46.298241 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 045e81ca74664ceba2ca1969f81f481f (DEFAULT_TABLE table=default.Table_e1fb587d58c340da84a5c8f0e0c1d7ee [id=e4bbb529da3345b9a8a0945ed303eb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.298422 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 350) took 2010 ms. Trace:
I20251211 08:57:46.298808 15299 rpcz_store.cc:276] 1211 08:57:44.288502 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.288886 (+ 384us) service_pool.cc:225] Handling call
1211 08:57:46.292789 (+2003903us) ts_tablet_manager.cc:1151] Deleting tablet 07f983604bfa474f9c9ce0c982d738fc
1211 08:57:46.298141 (+ 5352us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.298373 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1985848}
I20251211 08:57:46.299209 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045e81ca74664ceba2ca1969f81f481f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.299791 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 351) took 2011 ms. Trace:
I20251211 08:57:46.300050 15299 rpcz_store.cc:276] 1211 08:57:44.288774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.289086 (+ 312us) service_pool.cc:225] Handling call
1211 08:57:46.299544 (+2010458us) ts_tablet_manager.cc:1151] Deleting tablet f909d82e0136470a810e1063d6769868
1211 08:57:46.299575 (+ 31us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.299746 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1992404}
I20251211 08:57:46.301000 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 354) took 1858 ms. Trace:
I20251211 08:57:46.301252 15299 rpcz_store.cc:276] 1211 08:57:44.442742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.455056 (+ 12314us) service_pool.cc:225] Handling call
1211 08:57:46.300705 (+1845649us) ts_tablet_manager.cc:1151] Deleting tablet 5d8d84e594ea47f3bc9127cc587e8aa7
1211 08:57:46.300755 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.300959 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1840470}
I20251211 08:57:46.308552 14271 tablet_service.cc:1505] Processing CreateTablet for tablet b49ea29851d94622b4299e93804a0578 (DEFAULT_TABLE table=default.Table_2839940578ba47e88ac23a1850c356ed [id=6221ad49ac60429db5855f4e7e3f9ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.309471 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b49ea29851d94622b4299e93804a0578. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.310148 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
I20251211 08:57:46.318498 14274 tablet_service.cc:1505] Processing CreateTablet for tablet faa96580439242948a66bbf00aeb836a (DEFAULT_TABLE table=default.Table_7558ae666eb04c7b96e3d1151f22fbe6 [id=ee545203a1b4427a9a87f353e28db014]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.319410 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faa96580439242948a66bbf00aeb836a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:46.329813 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 170941us
I20251211 08:57:46.330565 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 4e813a4c73334c839bfe6af2119e259c (table default.Table_358b3598ddbe4cc2831ba06d3600ac09 [id=ffaa7c2a60924b0cbdc1254bfad98189]) successfully deleted
I20251211 08:57:46.331141 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 370) took 1325 ms. Trace:
W20251211 08:57:46.331456 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 4c95565ce3954cc598d7cbd45321d7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c95565ce3954cc598d7cbd45321d7c9 already in progress: creating tablet
I20251211 08:57:46.331462 15299 rpcz_store.cc:276] 1211 08:57:45.005692 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.011872 (+ 6180us) service_pool.cc:225] Handling call
1211 08:57:46.302741 (+1290869us) ts_tablet_manager.cc:1151] Deleting tablet 4631dd0bc0c144e69a31de80de39a76a
1211 08:57:46.330819 (+ 28078us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.331091 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290104}
I20251211 08:57:46.332341 15299 tablet_replica.cc:333] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:46.333043 15299 raft_consensus.cc:2243] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:46.333474 15299 raft_consensus.cc:2272] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:46.334996 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet ed0e90efe651499c929b1fb992712b96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed0e90efe651499c929b1fb992712b96 already in progress: creating tablet
W20251211 08:57:46.336238 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 07f983604bfa474f9c9ce0c982d738fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07f983604bfa474f9c9ce0c982d738fc already in progress: creating tablet
I20251211 08:57:46.342073 14722 tablet_bootstrap.cc:654] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:46.354655 14271 tablet_service.cc:1505] Processing CreateTablet for tablet 3567c7fe6104474f8d550cc0d2855ed6 (DEFAULT_TABLE table=default.Table_6bd043cbc5024c4c9881e15f002ce3cf [id=28663a7452044b3fae9842b2da59d8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.355613 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3567c7fe6104474f8d550cc0d2855ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.356040 14904 raft_consensus.cc:2243] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:46.358489 14904 pending_rounds.cc:70] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
I20251211 08:57:46.358664 14904 pending_rounds.cc:77] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256422073585664 op_type: NO_OP noop_request { }
I20251211 08:57:46.358994 14904 raft_consensus.cc:2889] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:46.359141 15299 ts_tablet_manager.cc:1916] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:46.359278 14904 raft_consensus.cc:2272] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:46.359534 14230 leader_election.cc:304] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:46.360677 14382 leader_election.cc:304] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
W20251211 08:57:46.361222 14382 raft_consensus.cc:2686] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251211 08:57:46.365171 15554 raft_consensus.cc:3060] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:46.182578 15030 raft_consensus.cc:359] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.371405 15030 raft_consensus.cc:385] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:46.185990 14905 raft_consensus.cc:2243] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:46.371659 15030 raft_consensus.cc:740] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:46.372320 14905 pending_rounds.cc:70] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:57:46.372241 15030 consensus_queue.cc:260] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.372785 14010 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet c6d419632b2a49e1975d4baa8ac85322
I20251211 08:57:46.373654 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 00e5c4bf71b44abca66b0e94995c533e
I20251211 08:57:46.373924 14010 catalog_manager.cc:6724] ReplaceTablet: tablet c6d419632b2a49e1975d4baa8ac85322 deleted and replaced by tablet 7b6cdf1adae34ad2908233ce7323cf65
I20251211 08:57:46.374272 15030 ts_tablet_manager.cc:1434] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.194s user 0.006s sys 0.000s
I20251211 08:57:46.375510 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet c6d419632b2a49e1975d4baa8ac85322 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b6cdf1adae34ad2908233ce7323cf65) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:46.376588 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 00e5c4bf71b44abca66b0e94995c533e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:46.317852 14576 tablet_service.cc:1505] Processing CreateTablet for tablet b49ea29851d94622b4299e93804a0578 (DEFAULT_TABLE table=default.Table_2839940578ba47e88ac23a1850c356ed [id=6221ad49ac60429db5855f4e7e3f9ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.379467 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b49ea29851d94622b4299e93804a0578. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.379779 15030 tablet_bootstrap.cc:492] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:46.380326 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 3567c7fe6104474f8d550cc0d2855ed6 (DEFAULT_TABLE table=default.Table_6bd043cbc5024c4c9881e15f002ce3cf [id=28663a7452044b3fae9842b2da59d8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.381135 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3567c7fe6104474f8d550cc0d2855ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.384521 14584 tablet_service.cc:1505] Processing CreateTablet for tablet a9077b6044f74571baaeb34b62377d66 (DEFAULT_TABLE table=default.Table_6f4cd1808b3d49929bca9414357073b5 [id=a0784f808dec44ff8a1b1eaa6437c9e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.384976 14587 tablet_service.cc:1505] Processing CreateTablet for tablet b6cc7d4eba584426ac4e0cfcd05216c6 (DEFAULT_TABLE table=default.Table_5ceedd3100314612ab5b31cbcede0571 [id=feb5d51303c64c5dab6401ea218d2425]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.385365 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9077b6044f74571baaeb34b62377d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.385803 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6cc7d4eba584426ac4e0cfcd05216c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.387322 14266 tablet_service.cc:1505] Processing CreateTablet for tablet a9077b6044f74571baaeb34b62377d66 (DEFAULT_TABLE table=default.Table_6f4cd1808b3d49929bca9414357073b5 [id=a0784f808dec44ff8a1b1eaa6437c9e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.388232 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9077b6044f74571baaeb34b62377d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.389529 15030 tablet_bootstrap.cc:654] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:46.391885 14265 tablet_service.cc:1505] Processing CreateTablet for tablet b6cc7d4eba584426ac4e0cfcd05216c6 (DEFAULT_TABLE table=default.Table_5ceedd3100314612ab5b31cbcede0571 [id=feb5d51303c64c5dab6401ea218d2425]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.392772 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6cc7d4eba584426ac4e0cfcd05216c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:46.395671 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f909d82e0136470a810e1063d6769868 because the tablet was not found. No further retry: Not found: Tablet not found: f909d82e0136470a810e1063d6769868
I20251211 08:57:46.395768 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 63492ba35e26450f9645379a6501d544 (DEFAULT_TABLE table=default.Table_034c882ef42046d58a86c1b90f5cdf85 [id=fa191130e71d4b9ca15a924f3454a94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.396631 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63492ba35e26450f9645379a6501d544. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:46.396649 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d8d84e594ea47f3bc9127cc587e8aa7 already in progress: creating tablet
I20251211 08:57:46.402603 14274 tablet_service.cc:1552] Processing DeleteTablet for tablet 00e5c4bf71b44abca66b0e94995c533e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:46.403223 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet c6d419632b2a49e1975d4baa8ac85322 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b6cdf1adae34ad2908233ce7323cf65) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:46.412777 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 045e81ca74664ceba2ca1969f81f481f (DEFAULT_TABLE table=default.Table_e1fb587d58c340da84a5c8f0e0c1d7ee [id=e4bbb529da3345b9a8a0945ed303eb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.413197 14423 tablet_service.cc:1505] Processing CreateTablet for tablet faa96580439242948a66bbf00aeb836a (DEFAULT_TABLE table=default.Table_7558ae666eb04c7b96e3d1151f22fbe6 [id=ee545203a1b4427a9a87f353e28db014]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.413692 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045e81ca74664ceba2ca1969f81f481f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.414165 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faa96580439242948a66bbf00aeb836a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.415097 14421 tablet_service.cc:1505] Processing CreateTablet for tablet b49ea29851d94622b4299e93804a0578 (DEFAULT_TABLE table=default.Table_2839940578ba47e88ac23a1850c356ed [id=6221ad49ac60429db5855f4e7e3f9ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.416117 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b49ea29851d94622b4299e93804a0578. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.424110 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c39cc816f4c438391bb4826ea1a508f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:46.435905 14544 leader_election.cc:343] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:46.436420 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 63492ba35e26450f9645379a6501d544 (DEFAULT_TABLE table=default.Table_034c882ef42046d58a86c1b90f5cdf85 [id=fa191130e71d4b9ca15a924f3454a94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.439466 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63492ba35e26450f9645379a6501d544. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.442297 14904 ts_tablet_manager.cc:1916] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:46.346931 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 126650us
I20251211 08:57:46.353675 14578 tablet_service.cc:1505] Processing CreateTablet for tablet faa96580439242948a66bbf00aeb836a (DEFAULT_TABLE table=default.Table_7558ae666eb04c7b96e3d1151f22fbe6 [id=ee545203a1b4427a9a87f353e28db014]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.446417 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faa96580439242948a66bbf00aeb836a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.372565 14905 pending_rounds.cc:77] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256420943982592 op_type: NO_OP noop_request { }
I20251211 08:57:46.449038 14905 raft_consensus.cc:2889] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:46.449378 14905 raft_consensus.cc:2272] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:46.451068 14905 ts_tablet_manager.cc:1916] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:46.267776 14714 tablet_bootstrap.cc:654] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:46.307534 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 045e81ca74664ceba2ca1969f81f481f (DEFAULT_TABLE table=default.Table_e1fb587d58c340da84a5c8f0e0c1d7ee [id=e4bbb529da3345b9a8a0945ed303eb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.454589 14418 tablet_service.cc:1505] Processing CreateTablet for tablet 3567c7fe6104474f8d550cc0d2855ed6 (DEFAULT_TABLE table=default.Table_6bd043cbc5024c4c9881e15f002ce3cf [id=28663a7452044b3fae9842b2da59d8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.455365 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045e81ca74664ceba2ca1969f81f481f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.455554 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3567c7fe6104474f8d550cc0d2855ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.457332 14273 tablet_service.cc:1505] Processing CreateTablet for tablet d25fcaffc12e479eadd0a6c33c73b6c7 (DEFAULT_TABLE table=default.Table_917836f5cc9542c1a13cd99b4456b033 [id=454751abe8cc49e78c8463b49aa2067c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.458166 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25fcaffc12e479eadd0a6c33c73b6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.459343 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_846ceafaf2c74f698b8077c65736e2cf" } new_table_name: "default.Table_b58bca5097ae4e3cab8c112bc4d17bbb" modify_external_catalogs: true
W20251211 08:57:46.468075 13999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20251211 08:57:46.471403 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet c6d419632b2a49e1975d4baa8ac85322 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b6cdf1adae34ad2908233ce7323cf65) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:46.472343 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 00e5c4bf71b44abca66b0e94995c533e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:46.473014 14007 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 90c8960c5bfa4a08905bd211654e4798 (table default.Table_d2367e8cf559401c8af423ce7425f5a8 [id=9f3e442a656a4a8080fbf847cb136b00]). Expected version 1 got 0
I20251211 08:57:46.473515 14426 tablet_service.cc:1505] Processing CreateTablet for tablet a9077b6044f74571baaeb34b62377d66 (DEFAULT_TABLE table=default.Table_6f4cd1808b3d49929bca9414357073b5 [id=a0784f808dec44ff8a1b1eaa6437c9e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.474345 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9077b6044f74571baaeb34b62377d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.481103 14904 ts_tablet_manager.cc:1929] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:46.481431 14904 log.cc:1199] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/9b5a4b6b1b28406c82e2acd9356b60d4
I20251211 08:57:46.482129 14904 ts_tablet_manager.cc:1950] T 9b5a4b6b1b28406c82e2acd9356b60d4 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:57:46.495260 14542 leader_election.cc:343] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:46.495800 14542 leader_election.cc:304] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:46.500628 14416 tablet_service.cc:1505] Processing CreateTablet for tablet 63492ba35e26450f9645379a6501d544 (DEFAULT_TABLE table=default.Table_034c882ef42046d58a86c1b90f5cdf85 [id=fa191130e71d4b9ca15a924f3454a94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.501606 14416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63492ba35e26450f9645379a6501d544. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.503983 14425 tablet_service.cc:1505] Processing CreateTablet for tablet b6cc7d4eba584426ac4e0cfcd05216c6 (DEFAULT_TABLE table=default.Table_5ceedd3100314612ab5b31cbcede0571 [id=feb5d51303c64c5dab6401ea218d2425]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.504947 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6cc7d4eba584426ac4e0cfcd05216c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.514427 15030 tablet_bootstrap.cc:492] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:46.515564 15030 ts_tablet_manager.cc:1403] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.136s user 0.010s sys 0.000s
I20251211 08:57:46.519186 14428 tablet_service.cc:1505] Processing CreateTablet for tablet d25fcaffc12e479eadd0a6c33c73b6c7 (DEFAULT_TABLE table=default.Table_917836f5cc9542c1a13cd99b4456b033 [id=454751abe8cc49e78c8463b49aa2067c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.520236 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25fcaffc12e479eadd0a6c33c73b6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.523053 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b7ff902a532d4e7697811e15bcafde66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:46.524521 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7ff902a532d4e7697811e15bcafde66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:46.524735 15512 raft_consensus.cc:493] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:46.525174 15512 raft_consensus.cc:515] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.526976 15512 leader_election.cc:290] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:46.528532 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c8960c5bfa4a08905bd211654e4798" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:46.533398 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 5bd3377efd83457f91c1bf05fba12cd1 (DEFAULT_TABLE table=default.Table_d2111e461ff04909b0f4f135f0470016 [id=ca4016b9ed9844a99b4139aff46823e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.534328 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd3377efd83457f91c1bf05fba12cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.534382 15593 raft_consensus.cc:2749] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:46.535053 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 5bd3377efd83457f91c1bf05fba12cd1 (DEFAULT_TABLE table=default.Table_d2111e461ff04909b0f4f135f0470016 [id=ca4016b9ed9844a99b4139aff46823e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.535477 14542 leader_election.cc:304] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:46.535490 15512 raft_consensus.cc:493] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:46.536017 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd3377efd83457f91c1bf05fba12cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.536154 15512 raft_consensus.cc:515] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.538802 15512 leader_election.cc:290] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:46.539744 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 102232us
W20251211 08:57:46.540311 14231 leader_election.cc:343] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 4eec7694087145d8a49dd93904f95980
W20251211 08:57:46.540375 14544 leader_election.cc:343] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:46.542575 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 4631dd0bc0c144e69a31de80de39a76a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4631dd0bc0c144e69a31de80de39a76a already in progress: creating tablet
I20251211 08:57:46.543396 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 9b5a4b6b1b28406c82e2acd9356b60d4 (table default.Table_baaa95f73b8742388a79269d898a5a17 [id=99243463ed7545ff80f928e6664a3748]) successfully deleted
I20251211 08:57:46.543444 14904 tablet_replica.cc:333] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:46.550109 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet faa96580439242948a66bbf00aeb836a
W20251211 08:57:46.551892 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f0cd7bb9fae4a2393578922fd359e6e consensus state has no leader
I20251211 08:57:46.552986 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet faa96580439242948a66bbf00aeb836a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:46 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:46.557646 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet faa96580439242948a66bbf00aeb836a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:46 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:46.559027 15593 raft_consensus.cc:2804] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:46.559379 15593 raft_consensus.cc:493] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:46.559839 14714 tablet_bootstrap.cc:492] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:46.559782 15593 raft_consensus.cc:3060] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:46.560372 14714 ts_tablet_manager.cc:1403] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.322s user 0.009s sys 0.000s
I20251211 08:57:46.562019 14580 tablet_service.cc:1505] Processing CreateTablet for tablet d25fcaffc12e479eadd0a6c33c73b6c7 (DEFAULT_TABLE table=default.Table_917836f5cc9542c1a13cd99b4456b033 [id=454751abe8cc49e78c8463b49aa2067c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.564493 14421 tablet_service.cc:1505] Processing CreateTablet for tablet 5bd3377efd83457f91c1bf05fba12cd1 (DEFAULT_TABLE table=default.Table_d2111e461ff04909b0f4f135f0470016 [id=ca4016b9ed9844a99b4139aff46823e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.577946 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25fcaffc12e479eadd0a6c33c73b6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.580022 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a2e57210dc9747d2926a04ef6cdeb10f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:46.580947 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 7f0cd7bb9fae4a2393578922fd359e6e (table default.Table_10d2cf5a95b9442da2fb73a50e9cbd8d [id=1e8b5ddeeea54c5f8e1951c184d77c27], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7f0cd7bb9fae4a2393578922fd359e6e consensus state has no leader
W20251211 08:57:46.581521 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2e57210dc9747d2926a04ef6cdeb10f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:46.568482 15593 raft_consensus.cc:515] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.582157 15581 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251211 08:57:46.595294 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4e15a6b7343e46c491019358d60e3589" } modify_external_catalogs: true
I20251211 08:57:46.595800 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4e15a6b7343e46c491019358d60e3589" } modify_external_catalogs: true
I20251211 08:57:46.596140 15593 leader_election.cc:290] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:46.585228 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 43583.6us
I20251211 08:57:46.603377 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ccfcacb5b294a628926d34cd31b137e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:46.603950 14604 raft_consensus.cc:3060] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:46.604883 14904 raft_consensus.cc:2243] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:46.605525 14904 raft_consensus.cc:2272] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:46.607396 14904 ts_tablet_manager.cc:1916] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:46.608042 14274 tablet_service.cc:1552] Processing DeleteTablet for tablet faa96580439242948a66bbf00aeb836a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:46 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:46.610618 14604 raft_consensus.cc:2468] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:46.613261 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c8960c5bfa4a08905bd211654e4798" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:46.614410 15554 raft_consensus.cc:2749] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:46.615199 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eec7694087145d8a49dd93904f95980" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:46.615756 14605 raft_consensus.cc:2393] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:46.616917 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_481bef424de1464b95f1e738b5986299"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:46.618346 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_481bef424de1464b95f1e738b5986299 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:46.622172 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd3377efd83457f91c1bf05fba12cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.586716 15030 raft_consensus.cc:359] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.626021 15030 raft_consensus.cc:385] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:46.626030 14230 leader_election.cc:304] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:46.626307 15030 raft_consensus.cc:740] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
W20251211 08:57:46.626567 14230 raft_consensus.cc:2686] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251211 08:57:46.632963 14230 leader_election.cc:304] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:46.633209 14714 raft_consensus.cc:359] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.633773 14714 raft_consensus.cc:385] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:46.633873 14230 leader_election.cc:304] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:46.634016 14714 raft_consensus.cc:740] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:46.634989 15512 raft_consensus.cc:2749] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:46.635484 15512 raft_consensus.cc:3060] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:46.642364 14722 tablet_bootstrap.cc:492] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:46.642760 14722 ts_tablet_manager.cc:1403] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.361s user 0.016s sys 0.019s
I20251211 08:57:46.644958 14722 raft_consensus.cc:359] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.645524 14722 raft_consensus.cc:385] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:46.645749 14722 raft_consensus.cc:740] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:46.646273 14722 consensus_queue.cc:260] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.650158 14722 ts_tablet_manager.cc:1434] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251211 08:57:46.653944 15547 raft_consensus.cc:493] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:46.654346 15547 raft_consensus.cc:515] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.656324 15547 leader_election.cc:290] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:46.660759 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f938359354b545cfbac560b7d7888b9f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:46.661072 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c39cc816f4c438391bb4826ea1a508f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:46.661550 14446 raft_consensus.cc:3060] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:46.662389 15299 ts_tablet_manager.cc:1929] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:46.662849 14722 tablet_bootstrap.cc:492] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:46.665277 15512 raft_consensus.cc:2749] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:46.666828 14013 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 45c7d841bd3643a799a8b96e82d6d675 (table default.Table_f1453eb82ce2412c8208000d8ba7f240 [id=f08b1a30da03451ba864cb8f8e6d199b]). Expected version 1 got 0
W20251211 08:57:46.663793 14385 leader_election.cc:343] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:46.671299 15299 log.cc:1199] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/3c2fb2d9564c43ee84b8dfcea8ec54b0
I20251211 08:57:46.672166 15299 ts_tablet_manager.cc:1950] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:46.675880 14013 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 578404732eef4e51b66b6aebffa87cdb (table default.Table_bf148c361a2346e88e4c9e4430b1abca [id=26c0a9d9b1f84372b18a54f045eb768c]). Expected version 4 got 3
I20251211 08:57:46.668129 14446 raft_consensus.cc:2468] T 8c39cc816f4c438391bb4826ea1a508f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:46.681483 14006 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_d2367e8cf559401c8af423ce7425f5a8 [id=9f3e442a656a4a8080fbf847cb136b00] alter complete (version 1)
I20251211 08:57:46.688066 14275 tablet_service.cc:1505] Processing CreateTablet for tablet f5de11a15c694423b266fea1cc2a7f99 (DEFAULT_TABLE table=default.Table_eb05ea73b5a7414e9914f98266240d00 [id=930f9ad3f13b4811b6268b4aff6ef3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.688956 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5de11a15c694423b266fea1cc2a7f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.692608 15613 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251211 08:57:46.694993 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e813a4c73334c839bfe6af2119e259c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:46.696578 14542 consensus_peers.cc:597] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e813a4c73334c839bfe6af2119e259c. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:46.699287 14542 leader_election.cc:304] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:46.699975 15561 raft_consensus.cc:2804] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:46.700362 15561 raft_consensus.cc:697] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:46.701014 15561 consensus_queue.cc:237] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:46.702358 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 68620.6us
I20251211 08:57:46.706634 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b462cf21e3420c8e323a4ac4f04abe with delete_type TABLET_DATA_DELETED (replaced by tablet 2ed430a2c82d498a9c3ca116eaac154e) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:46.707455 14298 raft_consensus.cc:3060] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:46.635015 15030 consensus_queue.cc:260] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.712016 15030 ts_tablet_manager.cc:1434] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.196s user 0.006s sys 0.000s
I20251211 08:57:46.715129 14905 ts_tablet_manager.cc:1929] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:46.715467 14905 log.cc:1199] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/dca342fd9de241b5884600199f06fbeb
I20251211 08:57:46.715361 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c39cc816f4c438391bb4826ea1a508f" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:46.716149 14905 ts_tablet_manager.cc:1950] T dca342fd9de241b5884600199f06fbeb P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:46.727210 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 371) took 1716 ms. Trace:
I20251211 08:57:46.728109 14010 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 9821822304c948f3bfcf5d52901371a4
I20251211 08:57:46.727782 14714 consensus_queue.cc:260] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:46.728350 14542 consensus_peers.cc:597] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2016208a33dc4a5cadd7ced8e85ccc65. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:46.733091 14583 tablet_service.cc:1505] Processing CreateTablet for tablet f5de11a15c694423b266fea1cc2a7f99 (DEFAULT_TABLE table=default.Table_eb05ea73b5a7414e9914f98266240d00 [id=930f9ad3f13b4811b6268b4aff6ef3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.735165 14576 tablet_service.cc:1505] Processing CreateTablet for tablet 20edf7ed321146a19a3ab6c69ba7cad6 (DEFAULT_TABLE table=default.Table_eacf82448cad4680a9e0ca3abe8ee5bb [id=e10c5a795eed48b7936f6f8cb02f1cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.736181 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab (DEFAULT_TABLE table=default.Table_1a66b3d9fba24b2e9415fb8743297c0f [id=78961c2045bf45739d3632c0abf6325d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.737183 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.737211 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 9821822304c948f3bfcf5d52901371a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:46 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:46.738657 14423 tablet_service.cc:1505] Processing CreateTablet for tablet 20edf7ed321146a19a3ab6c69ba7cad6 (DEFAULT_TABLE table=default.Table_eacf82448cad4680a9e0ca3abe8ee5bb [id=e10c5a795eed48b7936f6f8cb02f1cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.739889 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20edf7ed321146a19a3ab6c69ba7cad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.740787 15299 rpcz_store.cc:276] 1211 08:57:45.010559 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.012754 (+ 2195us) service_pool.cc:225] Handling call
1211 08:57:46.332199 (+1319445us) ts_tablet_manager.cc:1151] Deleting tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0
1211 08:57:46.332235 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.501667 (+169432us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:46.662189 (+160522us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:46.727159 (+ 64970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1318931}
I20251211 08:57:46.741982 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 372) took 1483 ms. Trace:
I20251211 08:57:46.742266 15299 rpcz_store.cc:276] 1211 08:57:45.258690 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.261933 (+ 3243us) service_pool.cc:225] Handling call
1211 08:57:46.741688 (+1479755us) ts_tablet_manager.cc:1151] Deleting tablet 01310095bca4479da24185576091b748
1211 08:57:46.741725 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.741936 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479252}
I20251211 08:57:46.743168 15299 tablet_replica.cc:333] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:46.743861 15299 raft_consensus.cc:2243] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:46.743793 14265 tablet_service.cc:1505] Processing CreateTablet for tablet 20edf7ed321146a19a3ab6c69ba7cad6 (DEFAULT_TABLE table=default.Table_eacf82448cad4680a9e0ca3abe8ee5bb [id=e10c5a795eed48b7936f6f8cb02f1cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.744395 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 (table default.Table_d8ac34f0cab345dd9b113ef1447ea361 [id=16e3304bc6774f7dab11c7ebd6eaa771]) successfully deleted
I20251211 08:57:46.745136 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20edf7ed321146a19a3ab6c69ba7cad6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:46.745229 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 01310095bca4479da24185576091b748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01310095bca4479da24185576091b748 already in progress: creating tablet
I20251211 08:57:46.745487 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20edf7ed321146a19a3ab6c69ba7cad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.746503 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f938359354b545cfbac560b7d7888b9f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:46.747169 14604 raft_consensus.cc:2468] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
W20251211 08:57:46.748723 14544 consensus_peers.cc:597] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b5a4b6b1b28406c82e2acd9356b60d4. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:46.751547 14418 tablet_service.cc:1505] Processing CreateTablet for tablet f5de11a15c694423b266fea1cc2a7f99 (DEFAULT_TABLE table=default.Table_eb05ea73b5a7414e9914f98266240d00 [id=930f9ad3f13b4811b6268b4aff6ef3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.755151 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_eed8ebdeeca94232a29625f6435c6ea0" } modify_external_catalogs: true
I20251211 08:57:46.759202 14010 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7158a7ed7bcb4153a059e26dbcd01a0e" } new_table_name: "default.Table_e2bf2dd74d94446aa1dd883c2a18ad68" modify_external_catalogs: true
I20251211 08:57:46.755694 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_eed8ebdeeca94232a29625f6435c6ea0" } modify_external_catalogs: true
I20251211 08:57:46.774271 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5de11a15c694423b266fea1cc2a7f99. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:46.775401 14544 leader_election.cc:343] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:46.777812 15617 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251211 08:57:46.761981 15547 raft_consensus.cc:493] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:46.784205 15547 raft_consensus.cc:515] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.784972 14004 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
I20251211 08:57:46.786128 15547 leader_election.cc:290] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:46.772687 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet 9821822304c948f3bfcf5d52901371a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:46 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:46.753621 14714 ts_tablet_manager.cc:1434] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.193s user 0.006s sys 0.000s
I20251211 08:57:46.797991 14714 tablet_bootstrap.cc:492] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:46.754246 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5de11a15c694423b266fea1cc2a7f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.769974 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 287) took 4104 ms. Trace:
I20251211 08:57:46.802503 14905 rpcz_store.cc:276] 1211 08:57:42.665500 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.665627 (+ 127us) service_pool.cc:225] Handling call
1211 08:57:46.164444 (+3498817us) ts_tablet_manager.cc:1151] Deleting tablet dca342fd9de241b5884600199f06fbeb
1211 08:57:46.165372 (+ 928us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.576948 (+411576us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:46.714880 (+137932us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:46.769873 (+ 54993us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3497094}
I20251211 08:57:46.803771 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet dca342fd9de241b5884600199f06fbeb (table default.Table_f627382c623a4f818caf527706e776ab [id=22f3369559b34f5aa54c92c1bb01c2cb]) successfully deleted
I20251211 08:57:46.804615 14905 tablet_replica.cc:333] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:46.805598 15030 tablet_bootstrap.cc:492] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:46.809755 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab (DEFAULT_TABLE table=default.Table_1a66b3d9fba24b2e9415fb8743297c0f [id=78961c2045bf45739d3632c0abf6325d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.810586 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.815963 14584 tablet_service.cc:1505] Processing CreateTablet for tablet f0ba9af8a85a44bfbafad59562916fea (DEFAULT_TABLE table=default.Table_bb1764135c31417987cdab15784dbea6 [id=ee42f539b8d147cca627aa0c63e2da42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.816846 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0ba9af8a85a44bfbafad59562916fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.824685 15622 tablet.cc:1722] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251211 08:57:46.812343 14416 tablet_service.cc:1505] Processing CreateTablet for tablet f0ba9af8a85a44bfbafad59562916fea (DEFAULT_TABLE table=default.Table_bb1764135c31417987cdab15784dbea6 [id=ee42f539b8d147cca627aa0c63e2da42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.827807 14416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0ba9af8a85a44bfbafad59562916fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.813340 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab (DEFAULT_TABLE table=default.Table_1a66b3d9fba24b2e9415fb8743297c0f [id=78961c2045bf45739d3632c0abf6325d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.831290 14905 raft_consensus.cc:2243] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:46.823328 14426 tablet_service.cc:1505] Processing CreateTablet for tablet e62ae5b540aa4680b4d18fee29c39e64 (DEFAULT_TABLE table=default.Table_70f3ebdd01dc4e75bc85ab29528865fc [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:46.832078 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 71051.3us
I20251211 08:57:46.832705 14382 leader_election.cc:304] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:46.781332 14722 tablet_bootstrap.cc:654] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:46.834942 15514 raft_consensus.cc:493] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:46.835347 15514 raft_consensus.cc:515] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.836046 15562 raft_consensus.cc:2804] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:46.836364 15562 raft_consensus.cc:493] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:46.836637 15562 raft_consensus.cc:3060] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:46.837174 15514 leader_election.cc:290] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:46.839110 14585 tablet_service.cc:1505] Processing CreateTablet for tablet e62ae5b540aa4680b4d18fee29c39e64 (DEFAULT_TABLE table=default.Table_70f3ebdd01dc4e75bc85ab29528865fc [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.839927 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62ae5b540aa4680b4d18fee29c39e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.842767 15562 raft_consensus.cc:515] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.801805 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e15483f539f24f1ba1b442663d66fd45" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:46.810711 14273 tablet_service.cc:1505] Processing CreateTablet for tablet f0ba9af8a85a44bfbafad59562916fea (DEFAULT_TABLE table=default.Table_bb1764135c31417987cdab15784dbea6 [id=ee42f539b8d147cca627aa0c63e2da42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.856148 14722 tablet_bootstrap.cc:492] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:46.856341 15562 leader_election.cc:290] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:46.856537 14722 ts_tablet_manager.cc:1403] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.198s user 0.008s sys 0.003s
I20251211 08:57:46.819696 14271 tablet_service.cc:1505] Processing CreateTablet for tablet e62ae5b540aa4680b4d18fee29c39e64 (DEFAULT_TABLE table=default.Table_70f3ebdd01dc4e75bc85ab29528865fc [id=67d1517b43294838bda420b792edb9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.823557 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 179e9a63def34ad09dff85c04e932e75 (DEFAULT_TABLE table=default.Table_30906dd001234b1aa252daaa6df361e0 [id=4ae4b99f5ceb497e9d988b93d638f7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.831451 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.860122 14722 raft_consensus.cc:359] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:46.860862 14385 leader_election.cc:343] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:46.861003 14722 raft_consensus.cc:385] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:46.861335 14722 raft_consensus.cc:740] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:46.861884 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62ae5b540aa4680b4d18fee29c39e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.862046 14722 consensus_queue.cc:260] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.864959 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_759bc12a73cc4cab9c90f127a2522939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:46.866292 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_759bc12a73cc4cab9c90f127a2522939 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:46.867601 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_91192655bac749ae8e0649eb05ab6179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:46.868880 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91192655bac749ae8e0649eb05ab6179 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:46.873692 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_299019253b214d92b585b6e128b24c3a" } new_table_name: "default.Table_6ce1c76d3d294fc4ba9a233d7470000e" modify_external_catalogs: true
W20251211 08:57:46.875560 14544 consensus_peers.cc:597] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b90b0c944174b2a82696e3fee806934. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:46.879731 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b9ae463731d4828afc02039e0d161b5" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:46.868872 14013 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
I20251211 08:57:46.883035 15299 raft_consensus.cc:2272] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:46.775869 15561 consensus_queue.cc:1048] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:46.737731 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 9821822304c948f3bfcf5d52901371a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:46 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:46.793079 15581 tablet.cc:1722] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251211 08:57:46.794950 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 49015c5d4f2b4d07b547a9df57c992eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:46.884989 15299 ts_tablet_manager.cc:1916] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:46.887295 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f938359354b545cfbac560b7d7888b9f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:46.888244 14905 pending_rounds.cc:70] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:57:46.888525 14905 pending_rounds.cc:77] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256421563760640 op_type: NO_OP noop_request { }
I20251211 08:57:46.888904 14905 raft_consensus.cc:2889] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:46.889297 14905 raft_consensus.cc:2272] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:46.891093 14905 ts_tablet_manager.cc:1916] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:46.894236 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b9ae463731d4828afc02039e0d161b5" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:46.884768 14008 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 4b9ae463731d4828afc02039e0d161b5 (table default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]). Expected version 1 got 0
I20251211 08:57:46.889144 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_49a263a32c9940a5b61d8a430ebb61a7 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 3 got 0
I20251211 08:57:46.897471 14008 catalog_manager.cc:5654] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:46.899693 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 179e9a63def34ad09dff85c04e932e75. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:46.900278 14542 leader_election.cc:343] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:46.900677 14385 leader_election.cc:343] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:46.907191 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62ae5b540aa4680b4d18fee29c39e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.911232 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0ba9af8a85a44bfbafad59562916fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.920379 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e15483f539f24f1ba1b442663d66fd45" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:46.920833 15612 raft_consensus.cc:493] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:46.921092 14604 raft_consensus.cc:2468] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:46.921254 15612 raft_consensus.cc:515] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.923352 15612 leader_election.cc:290] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:46.923718 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f938359354b545cfbac560b7d7888b9f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:46.924249 14604 raft_consensus.cc:3060] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:46.925403 14231 leader_election.cc:343] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 4e813a4c73334c839bfe6af2119e259c
I20251211 08:57:46.925509 15620 raft_consensus.cc:493] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:46.925910 15620 raft_consensus.cc:515] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:46.931825 14010 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 111adfd1ac1a42bdafa9416594e7e932 consensus state has no leader
I20251211 08:57:46.932451 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f32a1ccedd4f15bf8c79bc3f16e6bb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:46.935705 14714 tablet_bootstrap.cc:654] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:46.938997 15030 tablet_bootstrap.cc:654] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:46.940528 15612 raft_consensus.cc:493] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:46.941006 15612 raft_consensus.cc:515] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.945549 14382 leader_election.cc:304] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:46.946540 15623 raft_consensus.cc:2804] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:46.946717 15612 leader_election.cc:290] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:46.946926 15623 raft_consensus.cc:493] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:46.947285 15623 raft_consensus.cc:3060] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:46.947273 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 2e4c3a63cd7c4cc98a240e459e84a2ac (DEFAULT_TABLE table=default.Table_6b072b26d1564613a46376ccb5bb2635 [id=66855f515acd4e269f106704bf5b6908]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.949576 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4c3a63cd7c4cc98a240e459e84a2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.950649 14904 ts_tablet_manager.cc:1929] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:46.950992 14904 log.cc:1199] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/9ccfcacb5b294a628926d34cd31b137e
W20251211 08:57:46.952143 14231 leader_election.cc:343] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: f5b462cf21e3420c8e323a4ac4f04abe
W20251211 08:57:46.955533 14385 leader_election.cc:343] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:46.955858 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 2e4c3a63cd7c4cc98a240e459e84a2ac (DEFAULT_TABLE table=default.Table_6b072b26d1564613a46376ccb5bb2635 [id=66855f515acd4e269f106704bf5b6908]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.955596 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 179e9a63def34ad09dff85c04e932e75 (DEFAULT_TABLE table=default.Table_30906dd001234b1aa252daaa6df361e0 [id=4ae4b99f5ceb497e9d988b93d638f7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.956833 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4c3a63cd7c4cc98a240e459e84a2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.966517 14299 raft_consensus.cc:3060] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:46.967114 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 2e4c3a63cd7c4cc98a240e459e84a2ac (DEFAULT_TABLE table=default.Table_6b072b26d1564613a46376ccb5bb2635 [id=66855f515acd4e269f106704bf5b6908]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:46.966949 15030 tablet_bootstrap.cc:492] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:46.968616 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4c3a63cd7c4cc98a240e459e84a2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.969332 15030 ts_tablet_manager.cc:1403] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.225s user 0.009s sys 0.000s
I20251211 08:57:46.972183 15620 leader_election.cc:290] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:46.977177 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 179e9a63def34ad09dff85c04e932e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:46.977438 15030 raft_consensus.cc:359] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.978705 15030 raft_consensus.cc:385] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:46.978972 15030 raft_consensus.cc:740] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:46.979507 15030 consensus_queue.cc:260] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.983191 15581 tablet.cc:1717] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063: Skipping requested alter to schema version 4, tablet already version 4
I20251211 08:57:46.984099 15514 raft_consensus.cc:493] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:46.984479 15514 raft_consensus.cc:515] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:46.986249 15514 leader_election.cc:290] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:46.988044 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49015c5d4f2b4d07b547a9df57c992eb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:46.988435 14904 ts_tablet_manager.cc:1950] T 9ccfcacb5b294a628926d34cd31b137e P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:57:46.989279 14542 leader_election.cc:343] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:46.989306 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 340) took 1054 ms. Trace:
I20251211 08:57:46.991240 15030 ts_tablet_manager.cc:1434] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20251211 08:57:46.994680 15030 tablet_bootstrap.cc:492] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:46.991940 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 179e9a63def34ad09dff85c04e932e75 (DEFAULT_TABLE table=default.Table_30906dd001234b1aa252daaa6df361e0 [id=4ae4b99f5ceb497e9d988b93d638f7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.005200 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e813a4c73334c839bfe6af2119e259c" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:47.006513 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5b462cf21e3420c8e323a4ac4f04abe" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:47.007031 14607 raft_consensus.cc:2468] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:47.008008 15561 raft_consensus.cc:493] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31404d5f58e548519239569b150d975b)
I20251211 08:57:47.008562 15561 raft_consensus.cc:515] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } }
I20251211 08:57:46.989575 14904 rpcz_store.cc:276] 1211 08:57:45.934818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.934929 (+ 111us) service_pool.cc:225] Handling call
1211 08:57:46.543224 (+608295us) ts_tablet_manager.cc:1151] Deleting tablet 9ccfcacb5b294a628926d34cd31b137e
1211 08:57:46.543292 (+ 68us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:46.721528 (+178236us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:46.850633 (+129105us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:46.989255 (+138622us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":607787}
I20251211 08:57:47.010506 15561 leader_election.cc:290] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:47.010483 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 9ccfcacb5b294a628926d34cd31b137e (table default.Table_93954115dc5d4d2283faa424e83608b0 [id=cb30302a7db04098ae9a5a92ce055d40]) successfully deleted
I20251211 08:57:47.011253 14904 tablet_replica.cc:333] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b: stopping tablet replica
W20251211 08:57:47.011798 14542 leader_election.cc:343] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 593df63696a04f85b00c301d93748e80
I20251211 08:57:47.002630 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 179e9a63def34ad09dff85c04e932e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.013388 15030 tablet_bootstrap.cc:654] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:47.044026 15623 raft_consensus.cc:515] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.046998 15030 tablet_bootstrap.cc:492] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:47.047370 15030 ts_tablet_manager.cc:1403] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.003s
I20251211 08:57:47.050477 15030 raft_consensus.cc:359] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.051086 15030 raft_consensus.cc:385] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:47.051352 15030 raft_consensus.cc:740] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:47.052110 15030 consensus_queue.cc:260] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.052718 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_952b121ba0144ec3975634134ac3c07d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:47.054123 15030 ts_tablet_manager.cc:1434] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
W20251211 08:57:47.054312 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_952b121ba0144ec3975634134ac3c07d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:47.054929 15030 tablet_bootstrap.cc:492] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:47.058949 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7343d67071c54141af5a57615f82221e" } modify_external_catalogs: true
I20251211 08:57:47.061031 14722 ts_tablet_manager.cc:1434] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.204s user 0.007s sys 0.000s
I20251211 08:57:47.061982 14722 tablet_bootstrap.cc:492] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:47.066424 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_3b53cda0501e4b3fa6c51534dc6fa05e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:47.068231 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b53cda0501e4b3fa6c51534dc6fa05e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:47.069054 14274 tablet_service.cc:1505] Processing CreateTablet for tablet be316db7fcdd4ea2871079894900114e (DEFAULT_TABLE table=default.Table_e614e9972e50436aa5ad310e68a914fb [id=333acb14b43e416690c484709559944e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.070035 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be316db7fcdd4ea2871079894900114e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.070508 15030 tablet_bootstrap.cc:654] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:47.074985 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 145096302ca24d7e96f2fec45960b18a
I20251211 08:57:47.077215 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 145096302ca24d7e96f2fec45960b18a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:46 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:47.079496 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7343d67071c54141af5a57615f82221e" } modify_external_catalogs: true
I20251211 08:57:47.085800 14584 tablet_service.cc:1505] Processing CreateTablet for tablet be316db7fcdd4ea2871079894900114e (DEFAULT_TABLE table=default.Table_e614e9972e50436aa5ad310e68a914fb [id=333acb14b43e416690c484709559944e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.086759 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be316db7fcdd4ea2871079894900114e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.094269 15030 tablet_bootstrap.cc:492] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:47.094696 15030 ts_tablet_manager.cc:1403] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.007s
I20251211 08:57:47.097337 15030 raft_consensus.cc:359] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.103634 15030 raft_consensus.cc:385] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:47.103892 15030 raft_consensus.cc:740] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:47.104414 15030 consensus_queue.cc:260] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.106570 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet 145096302ca24d7e96f2fec45960b18a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:46 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:47.109076 15030 ts_tablet_manager.cc:1434] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20251211 08:57:47.109933 15030 tablet_bootstrap.cc:492] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:47.109762 14265 tablet_service.cc:1505] Processing CreateTablet for tablet aadc3c1ecd974ae3a0d491bd209047e6 (DEFAULT_TABLE table=default.Table_bed4cef3d74f4af8acaa18fed506bdea [id=a299e437efaf4d8faa2a1ab3087540c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.110914 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadc3c1ecd974ae3a0d491bd209047e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.114830 14904 raft_consensus.cc:2243] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:47.115254 14904 raft_consensus.cc:2272] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:47.115871 14275 tablet_service.cc:1505] Processing CreateTablet for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 (DEFAULT_TABLE table=default.Table_5df60df88ddd40deb8310ab9f413aab5 [id=baccb991ba0342118a78a2617bfb4c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.117120 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.119056 14578 tablet_service.cc:1505] Processing CreateTablet for tablet aadc3c1ecd974ae3a0d491bd209047e6 (DEFAULT_TABLE table=default.Table_bed4cef3d74f4af8acaa18fed506bdea [id=a299e437efaf4d8faa2a1ab3087540c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.120246 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadc3c1ecd974ae3a0d491bd209047e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.130702 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 643295fd5a4a484790766fdf863a57e0 (DEFAULT_TABLE table=default.Table_f819a255bf2a4b3ebcddaa81435bc4e8 [id=275b9c3ffa7c43cf8f583c0ff11c0372]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.132030 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643295fd5a4a484790766fdf863a57e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.133706 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 643295fd5a4a484790766fdf863a57e0 (DEFAULT_TABLE table=default.Table_f819a255bf2a4b3ebcddaa81435bc4e8 [id=275b9c3ffa7c43cf8f583c0ff11c0372]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.134562 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643295fd5a4a484790766fdf863a57e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.135373 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 960bc7a27fa04f2c9f398a67d2c93266 (DEFAULT_TABLE table=default.Table_80758110ce794260af2a551849bf4896 [id=277e1a7e34f54c16b29a738c17f75da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.136498 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960bc7a27fa04f2c9f398a67d2c93266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.137152 15634 tablet.cc:1722] T 90c8960c5bfa4a08905bd211654e4798 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:47.139591 14904 ts_tablet_manager.cc:1916] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:47.140591 15030 tablet_bootstrap.cc:654] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:47.148183 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_84155c7aff0d471687b30b86e434adf6" } new_table_name: "default.Table_cee1868d74ac4977b764abe19767825a" modify_external_catalogs: true
I20251211 08:57:47.156778 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 960bc7a27fa04f2c9f398a67d2c93266 (DEFAULT_TABLE table=default.Table_80758110ce794260af2a551849bf4896 [id=277e1a7e34f54c16b29a738c17f75da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.158056 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960bc7a27fa04f2c9f398a67d2c93266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.180401 14722 tablet_bootstrap.cc:654] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:47.184748 15030 tablet_bootstrap.cc:492] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:47.185120 15030 ts_tablet_manager.cc:1403] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.075s user 0.009s sys 0.008s
I20251211 08:57:47.193953 15030 raft_consensus.cc:359] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.194496 15030 raft_consensus.cc:385] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:47.194720 15030 raft_consensus.cc:740] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:47.199810 15030 consensus_queue.cc:260] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.201617 15030 ts_tablet_manager.cc:1434] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20251211 08:57:47.227922 15637 tablet.cc:1722] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251211 08:57:47.239804 14009 master_service.cc:954] ReplaceTablet: received request to replace tablet f938359354b545cfbac560b7d7888b9f from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:47.246392 14904 ts_tablet_manager.cc:1929] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:47.250999 15561 raft_consensus.cc:493] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:47.251407 15561 raft_consensus.cc:515] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.252893 14904 log.cc:1199] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/f909d82e0136470a810e1063d6769868
I20251211 08:57:47.254650 15561 leader_election.cc:290] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:47.255486 14904 ts_tablet_manager.cc:1950] T f909d82e0136470a810e1063d6769868 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:47.259133 15515 raft_consensus.cc:493] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:47.259629 15515 raft_consensus.cc:515] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.261842 15515 leader_election.cc:290] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:47.283586 14012 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_1a66b3d9fba24b2e9415fb8743297c0f" } modify_external_catalogs: true
I20251211 08:57:47.284070 14012 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_1a66b3d9fba24b2e9415fb8743297c0f" } modify_external_catalogs: true
I20251211 08:57:47.287551 15030 tablet_bootstrap.cc:492] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:47.293217 15030 tablet_bootstrap.cc:654] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:47.297309 14722 tablet_bootstrap.cc:492] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:47.297901 14722 ts_tablet_manager.cc:1403] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.236s user 0.012s sys 0.021s
I20251211 08:57:47.299787 14722 raft_consensus.cc:359] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.301389 14604 raft_consensus.cc:2468] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:47.302868 14722 raft_consensus.cc:385] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:47.303164 14722 raft_consensus.cc:740] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:47.303896 14722 consensus_queue.cc:260] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.306180 14722 ts_tablet_manager.cc:1434] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
W20251211 08:57:47.308795 14544 leader_election.cc:343] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:47.309266 14544 leader_election.cc:304] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:47.310056 15514 raft_consensus.cc:2749] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:47.317075 14580 tablet_service.cc:1505] Processing CreateTablet for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 (DEFAULT_TABLE table=default.Table_5df60df88ddd40deb8310ab9f413aab5 [id=baccb991ba0342118a78a2617bfb4c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.317920 15030 tablet_bootstrap.cc:492] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:47.318136 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.318277 15030 ts_tablet_manager.cc:1403] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.004s
I20251211 08:57:47.320288 15030 raft_consensus.cc:359] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.320837 15030 raft_consensus.cc:385] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:47.051465 15622 tablet.cc:1717] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256: Skipping requested alter to schema version 4, tablet already version 4
I20251211 08:57:47.083986 14422 tablet_service.cc:1505] Processing CreateTablet for tablet be316db7fcdd4ea2871079894900114e (DEFAULT_TABLE table=default.Table_e614e9972e50436aa5ad310e68a914fb [id=333acb14b43e416690c484709559944e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.340476 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be316db7fcdd4ea2871079894900114e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.340983 15623 leader_election.cc:290] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:47.084235 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 145096302ca24d7e96f2fec45960b18a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:46 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:47.101306 14417 tablet_service.cc:1505] Processing CreateTablet for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 (DEFAULT_TABLE table=default.Table_5df60df88ddd40deb8310ab9f413aab5 [id=baccb991ba0342118a78a2617bfb4c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.344363 14417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.104106 14427 tablet_service.cc:1505] Processing CreateTablet for tablet aadc3c1ecd974ae3a0d491bd209047e6 (DEFAULT_TABLE table=default.Table_bed4cef3d74f4af8acaa18fed506bdea [id=a299e437efaf4d8faa2a1ab3087540c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.152477 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 643295fd5a4a484790766fdf863a57e0 (DEFAULT_TABLE table=default.Table_f819a255bf2a4b3ebcddaa81435bc4e8 [id=275b9c3ffa7c43cf8f583c0ff11c0372]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.155627 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 960bc7a27fa04f2c9f398a67d2c93266 (DEFAULT_TABLE table=default.Table_80758110ce794260af2a551849bf4896 [id=277e1a7e34f54c16b29a738c17f75da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:47.254249 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ab48f6288494005ad42208e2e97ddf2" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:47.264046 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f7775994f684326b6312c802f6b084d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:47.348246 14542 leader_election.cc:343] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:47.349201 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadc3c1ecd974ae3a0d491bd209047e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.351238 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960bc7a27fa04f2c9f398a67d2c93266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.355161 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643295fd5a4a484790766fdf863a57e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:47.373697 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e15483f539f24f1ba1b442663d66fd45" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:47.374419 14298 raft_consensus.cc:3060] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:47.382256 14298 raft_consensus.cc:2468] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
W20251211 08:57:47.427083 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 121068us
I20251211 08:57:47.321075 15030 raft_consensus.cc:740] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
W20251211 08:57:47.400053 14576 consensus_meta.cc:287] T 20edf7ed321146a19a3ab6c69ba7cad6 P c5e624bcfd154cd586ef1dedb9940063: Time spent flushing consensus metadata: real 0.558s user 0.006s sys 0.000s
W20251211 08:57:47.406904 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 94465.5us
I20251211 08:57:47.436955 14230 leader_election.cc:304] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:47.437431 14382 leader_election.cc:304] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:47.437880 14230 leader_election.cc:304] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:47.439651 15612 raft_consensus.cc:2749] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:47.440204 15612 raft_consensus.cc:2804] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:47.440513 15612 raft_consensus.cc:493] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:47.440780 15612 raft_consensus.cc:3060] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:47.441738 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49015c5d4f2b4d07b547a9df57c992eb" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:47.432181 15030 consensus_queue.cc:260] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.445760 15030 ts_tablet_manager.cc:1434] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.127s user 0.005s sys 0.000s
W20251211 08:57:47.432727 14542 leader_election.cc:343] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:47.450028 15612 raft_consensus.cc:515] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.453634 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ab48f6288494005ad42208e2e97ddf2" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:47.455235 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f7775994f684326b6312c802f6b084d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:47.458072 14007 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_49a263a32c9940a5b61d8a430ebb61a7 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 3 got 0
W20251211 08:57:47.459569 14231 leader_election.cc:343] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: f5b462cf21e3420c8e323a4ac4f04abe
I20251211 08:57:47.469341 15612 leader_election.cc:290] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:47.470275 15617 tablet.cc:1717] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b: Skipping requested alter to schema version 4, tablet already version 4
I20251211 08:57:47.484565 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 341) took 1530 ms. Trace:
I20251211 08:57:47.484810 14904 rpcz_store.cc:276] 1211 08:57:45.954289 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.954394 (+ 105us) service_pool.cc:225] Handling call
1211 08:57:47.011126 (+1056732us) ts_tablet_manager.cc:1151] Deleting tablet f909d82e0136470a810e1063d6769868
1211 08:57:47.011161 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:47.200833 (+189672us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.246196 (+ 45363us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.484518 (+238322us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":102750,"tablet-delete.queue_time_us":1056323}
I20251211 08:57:47.485715 14904 tablet_replica.cc:333] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:47.486320 14904 raft_consensus.cc:2243] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251211 08:57:47.486896 14004 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d69451d9e49740bb9af84d39d157c2f0 consensus state has no leader
I20251211 08:57:47.487368 14722 tablet_bootstrap.cc:492] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:47.490262 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 121233us
I20251211 08:57:47.486676 14904 raft_consensus.cc:2272] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:47.491114 14385 leader_election.cc:304] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:47.492249 14904 ts_tablet_manager.cc:1916] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:47.492264 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet f909d82e0136470a810e1063d6769868 (table default.Table_c15367b621c144348ec8511d2947fec6 [id=ea01e214ed314a26a0de20d34e37863b]) successfully deleted
W20251211 08:57:47.492383 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 55047.8us
I20251211 08:57:47.500174 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b3c68d73c7154f10856f5fe4e01e3e8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:47.501602 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3c68d73c7154f10856f5fe4e01e3e8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:47.502933 14542 consensus_peers.cc:597] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:47.503343 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f32a1ccedd4f15bf8c79bc3f16e6bb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:47.503947 14604 raft_consensus.cc:2468] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:47.509842 14904 ts_tablet_manager.cc:1929] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:47.510239 14904 log.cc:1199] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/4eec7694087145d8a49dd93904f95980
I20251211 08:57:47.510881 14904 ts_tablet_manager.cc:1950] T 4eec7694087145d8a49dd93904f95980 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:47.512796 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e15483f539f24f1ba1b442663d66fd45" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:47.519709 14604 raft_consensus.cc:3060] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:47.522481 15547 raft_consensus.cc:2804] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:47.523553 15620 raft_consensus.cc:2804] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:47.529508 14604 raft_consensus.cc:2468] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:47.541560 15030 tablet_bootstrap.cc:492] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:47.541831 15514 raft_consensus.cc:493] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:47.542184 15514 raft_consensus.cc:515] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.544693 15514 leader_election.cc:290] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:47.546352 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98d14ba15dcd47f8b9e8035b3c7d477c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:47.547495 14542 leader_election.cc:343] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:47.547720 15030 tablet_bootstrap.cc:654] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:47.548260 15514 raft_consensus.cc:493] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:47.548681 15514 raft_consensus.cc:515] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.553037 14905 ts_tablet_manager.cc:1929] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:47.553431 14905 log.cc:1199] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/c53035e83710403bac3a2b401c8c97ad
I20251211 08:57:47.559873 14010 master_service.cc:954] ReplaceTablet: received request to replace tablet 14fc87cb76024b5b8227dfe9939e1c89 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:47.568832 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 51c3f5eaef784e868e565f81bd51a5e4
W20251211 08:57:47.568151 14542 leader_election.cc:343] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 4eec7694087145d8a49dd93904f95980
I20251211 08:57:47.568848 15514 leader_election.cc:290] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:47.579679 14905 ts_tablet_manager.cc:1950] T c53035e83710403bac3a2b401c8c97ad P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:47.576745 14576 tablet_service.cc:1552] Processing DeleteTablet for tablet 51c3f5eaef784e868e565f81bd51a5e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:47.569267 15646 raft_consensus.cc:493] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:47.580150 15646 raft_consensus.cc:515] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.582314 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 288) took 4915 ms. Trace:
I20251211 08:57:47.582787 14905 rpcz_store.cc:276] 1211 08:57:42.666403 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.666501 (+ 98us) service_pool.cc:225] Handling call
1211 08:57:46.804476 (+4137975us) ts_tablet_manager.cc:1151] Deleting tablet c53035e83710403bac3a2b401c8c97ad
1211 08:57:46.804515 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:47.455899 (+651384us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.552786 (+ 96887us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.582203 (+ 29417us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82270,"tablet-delete.queue_time_us":4137548}
W20251211 08:57:47.583601 14542 leader_election.cc:343] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: f909d82e0136470a810e1063d6769868
I20251211 08:57:47.583904 15646 leader_election.cc:290] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:47.595098 14905 tablet_replica.cc:333] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:47.597003 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet c53035e83710403bac3a2b401c8c97ad (table default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]) successfully deleted
W20251211 08:57:47.598263 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d69451d9e49740bb9af84d39d157c2f0 (table default.Table_6ce1c76d3d294fc4ba9a233d7470000e [id=a70a9aab34f347a698c850cb21298eb1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d69451d9e49740bb9af84d39d157c2f0 consensus state has no leader
I20251211 08:57:47.609902 15514 consensus_queue.cc:1048] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:47.616088 14905 raft_consensus.cc:2243] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:47.616652 14905 raft_consensus.cc:2272] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:47.618196 14905 ts_tablet_manager.cc:1916] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:47.634975 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2530df4933249e19d96759910a47673 consensus state has no leader
I20251211 08:57:47.642345 14722 tablet_bootstrap.cc:654] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:47.646478 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 342) took 1629 ms. Trace:
I20251211 08:57:47.646885 14904 rpcz_store.cc:276] 1211 08:57:46.017090 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.017747 (+ 657us) service_pool.cc:225] Handling call
1211 08:57:47.485592 (+1467845us) ts_tablet_manager.cc:1151] Deleting tablet 4eec7694087145d8a49dd93904f95980
1211 08:57:47.485626 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:47.503195 (+ 17569us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.509674 (+ 6479us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.646409 (+136735us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1467290}
I20251211 08:57:47.647998 14904 tablet_replica.cc:333] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:47.648696 14904 raft_consensus.cc:2243] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:47.649509 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 4eec7694087145d8a49dd93904f95980 (table default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]) successfully deleted
I20251211 08:57:47.655036 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_1c1540b4735a45c39b48fe0c26d9b89b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:47.656461 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c1540b4735a45c39b48fe0c26d9b89b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:47.657153 14904 raft_consensus.cc:2272] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:47.658886 14904 ts_tablet_manager.cc:1916] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:47.665839 14905 ts_tablet_manager.cc:1929] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:47.666946 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet 51c3f5eaef784e868e565f81bd51a5e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:47.635880 14417 tablet_service.cc:1552] Processing DeleteTablet for tablet 51c3f5eaef784e868e565f81bd51a5e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:47.673406 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet f938359354b545cfbac560b7d7888b9f
I20251211 08:57:47.637854 15620 raft_consensus.cc:697] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:47.674460 14009 catalog_manager.cc:6724] ReplaceTablet: tablet f938359354b545cfbac560b7d7888b9f deleted and replaced by tablet 361efe27fde244289c55f203e6c420e8
I20251211 08:57:47.678696 14417 tablet_service.cc:1552] Processing DeleteTablet for tablet f938359354b545cfbac560b7d7888b9f with delete_type TABLET_DATA_DELETED (replaced by tablet 361efe27fde244289c55f203e6c420e8) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:47.682303 14416 consensus_meta.cc:287] T f0ba9af8a85a44bfbafad59562916fea P b276f77f87254a5d909c5ab940a39256: Time spent flushing consensus metadata: real 0.669s user 0.001s sys 0.006s
W20251211 08:57:47.685704 14423 consensus_meta.cc:287] T 20edf7ed321146a19a3ab6c69ba7cad6 P b276f77f87254a5d909c5ab940a39256: Time spent flushing consensus metadata: real 0.827s user 0.008s sys 0.002s
I20251211 08:57:47.687999 14905 log.cc:1199] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/8ad600554c0946e0bcb0da6a42be2ffd
W20251211 08:57:47.690652 14418 consensus_meta.cc:287] T f5de11a15c694423b266fea1cc2a7f99 P b276f77f87254a5d909c5ab940a39256: Time spent flushing consensus metadata: real 0.649s user 0.007s sys 0.000s
I20251211 08:57:47.691046 14905 ts_tablet_manager.cc:1950] T 8ad600554c0946e0bcb0da6a42be2ffd P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:47.691309 15620 consensus_queue.cc:237] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.695446 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_917836f5cc9542c1a13cd99b4456b033" } new_table_name: "default.Table_987db3ee6e524ece8104a807609e75f0" modify_external_catalogs: true
I20251211 08:57:47.702507 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 8c1e2d6ba1014d00a6df8c6353cff3ab
I20251211 08:57:47.708737 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:47.709962 14265 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:47.711086 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:47.719596 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet f938359354b545cfbac560b7d7888b9f with delete_type TABLET_DATA_DELETED (replaced by tablet 361efe27fde244289c55f203e6c420e8) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:47.704010 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 289) took 5036 ms. Trace:
I20251211 08:57:47.720046 14905 rpcz_store.cc:276] 1211 08:57:42.667186 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.667292 (+ 106us) service_pool.cc:225] Handling call
1211 08:57:47.594983 (+4927691us) ts_tablet_manager.cc:1151] Deleting tablet 8ad600554c0946e0bcb0da6a42be2ffd
1211 08:57:47.595015 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:47.634698 (+ 39683us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.665609 (+ 30911us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.703956 (+ 38347us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8911,"tablet-delete.queue_time_us":4927277}
I20251211 08:57:47.721311 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 8ad600554c0946e0bcb0da6a42be2ffd (table default.Table_1eb40c19ca894f47ba26da25462c5c3a [id=62e4ec0c216b41eebee5490eaba6ef5f]) successfully deleted
I20251211 08:57:47.722482 14714 tablet_bootstrap.cc:492] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:47.722940 14714 ts_tablet_manager.cc:1403] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.925s user 0.013s sys 0.003s
I20251211 08:57:47.725121 14905 tablet_replica.cc:333] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:47.725862 14905 raft_consensus.cc:2243] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:47.726588 14905 raft_consensus.cc:2272] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:47.727236 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet f938359354b545cfbac560b7d7888b9f with delete_type TABLET_DATA_DELETED (replaced by tablet 361efe27fde244289c55f203e6c420e8) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:47.727725 15547 raft_consensus.cc:697] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:47.728492 14905 ts_tablet_manager.cc:1916] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:47.728426 15547 consensus_queue.cc:237] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.732136 15030 tablet_bootstrap.cc:492] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:47.733016 15030 ts_tablet_manager.cc:1403] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.192s user 0.011s sys 0.001s
I20251211 08:57:47.739041 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5b462cf21e3420c8e323a4ac4f04abe" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:47.739476 14607 raft_consensus.cc:3060] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:47.743332 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_bbb3bbff67a045babf963ea77831cda0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:47.744608 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbb3bbff67a045babf963ea77831cda0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:47.745740 14607 raft_consensus.cc:2468] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:47.782044 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_0c2ec37766b24f539c7ff1e0aa4ac9c0" } modify_external_catalogs: true
I20251211 08:57:47.782606 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_0c2ec37766b24f539c7ff1e0aa4ac9c0" } modify_external_catalogs: true
I20251211 08:57:47.806665 14382 leader_election.cc:304] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:47.812584 14722 tablet_bootstrap.cc:492] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:47.813005 14722 ts_tablet_manager.cc:1403] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.326s user 0.010s sys 0.005s
I20251211 08:57:47.823020 15648 raft_consensus.cc:2804] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:47.823417 15648 raft_consensus.cc:493] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:47.823737 15648 raft_consensus.cc:3060] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:47.814553 14905 ts_tablet_manager.cc:1929] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:47.834808 14905 log.cc:1199] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/9b5a4b6b1b28406c82e2acd9356b60d4
I20251211 08:57:47.834867 15648 raft_consensus.cc:515] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.838203 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_c0e87778556043e9ad8242434ab127c3" } new_table_name: "default.Table_5f90244381bf4b53ab7d861a768e6cd9" modify_external_catalogs: true
I20251211 08:57:47.840708 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f32a1ccedd4f15bf8c79bc3f16e6bb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:47.841272 14298 raft_consensus.cc:3060] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:47.835642 14905 ts_tablet_manager.cc:1950] T 9b5a4b6b1b28406c82e2acd9356b60d4 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:47.847550 15030 raft_consensus.cc:359] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.848220 15030 raft_consensus.cc:385] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:47.848487 15030 raft_consensus.cc:740] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:47.849215 15030 consensus_queue.cc:260] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.854843 15648 leader_election.cc:290] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:47.634088 15299 ts_tablet_manager.cc:1929] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:47.863307 15299 log.cc:1199] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/7b90b0c944174b2a82696e3fee806934
I20251211 08:57:47.864094 15299 ts_tablet_manager.cc:1950] T 7b90b0c944174b2a82696e3fee806934 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:47.865983 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 377) took 2289 ms. Trace:
I20251211 08:57:47.880461 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98d14ba15dcd47f8b9e8035b3c7d477c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:47.883342 14230 leader_election.cc:304] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:47.884189 15612 raft_consensus.cc:2804] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:47.884583 15612 raft_consensus.cc:697] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:47.885294 15612 consensus_queue.cc:237] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.914240 14298 raft_consensus.cc:2468] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:47.916375 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_bed4cef3d74f4af8acaa18fed506bdea" } modify_external_catalogs: true
I20251211 08:57:47.916909 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_bed4cef3d74f4af8acaa18fed506bdea" } modify_external_catalogs: true
I20251211 08:57:47.919113 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7ad765fbc3eb42568ff4edb8ae8223a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:47.920460 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ad765fbc3eb42568ff4edb8ae8223a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:47.932915 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d25fcaffc12e479eadd0a6c33c73b6c7 consensus state has no leader
I20251211 08:57:47.939173 15030 ts_tablet_manager.cc:1434] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.206s user 0.006s sys 0.000s
I20251211 08:57:47.941726 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4d9c098623e84435972e3c148a08b83e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:47.942058 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 290) took 5274 ms. Trace:
I20251211 08:57:47.942956 15646 raft_consensus.cc:493] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251211 08:57:47.943240 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d9c098623e84435972e3c148a08b83e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:47.943667 15646 raft_consensus.cc:515] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.947340 15646 leader_election.cc:290] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:47.942389 14905 rpcz_store.cc:276] 1211 08:57:42.668019 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.670875 (+ 2856us) service_pool.cc:225] Handling call
1211 08:57:47.724993 (+5054118us) ts_tablet_manager.cc:1151] Deleting tablet 9b5a4b6b1b28406c82e2acd9356b60d4
1211 08:57:47.725028 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:47.791503 (+ 66475us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.814338 (+ 22835us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.941986 (+127648us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22724,"tablet-delete.queue_time_us":5053736}
I20251211 08:57:47.949545 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 294) took 5174 ms. Trace:
I20251211 08:57:47.949823 14905 rpcz_store.cc:276] 1211 08:57:42.775277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.775462 (+ 185us) service_pool.cc:225] Handling call
1211 08:57:47.949226 (+5173764us) ts_tablet_manager.cc:1151] Deleting tablet f56fd93aeac6435ea84388ce05580113
1211 08:57:47.949269 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:47.949493 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5173210}
I20251211 08:57:47.951098 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 1a2dcb9ac62e4aac885d638f4e2b072e
I20251211 08:57:47.951354 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 9b5a4b6b1b28406c82e2acd9356b60d4 (table default.Table_baaa95f73b8742388a79269d898a5a17 [id=99243463ed7545ff80f928e6664a3748]) successfully deleted
W20251211 08:57:47.951932 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet f56fd93aeac6435ea84388ce05580113 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f56fd93aeac6435ea84388ce05580113 already in progress: creating tablet
I20251211 08:57:47.954226 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a2dcb9ac62e4aac885d638f4e2b072e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:47.956022 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 293) took 5184 ms. Trace:
I20251211 08:57:47.956396 14905 rpcz_store.cc:276] 1211 08:57:42.771925 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.772063 (+ 138us) service_pool.cc:225] Handling call
1211 08:57:47.950611 (+5178548us) ts_tablet_manager.cc:1151] Deleting tablet 2ed430a2c82d498a9c3ca116eaac154e
1211 08:57:47.950651 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:47.955972 (+ 5321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5174124}
W20251211 08:57:47.957558 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 2ed430a2c82d498a9c3ca116eaac154e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ed430a2c82d498a9c3ca116eaac154e already in progress: creating tablet
I20251211 08:57:47.958424 14905 tablet_replica.cc:333] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:47.959501 14905 raft_consensus.cc:2243] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:47.960085 14905 raft_consensus.cc:2272] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:47.962093 14905 ts_tablet_manager.cc:1916] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:47.968300 14544 consensus_peers.cc:597] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2016208a33dc4a5cadd7ced8e85ccc65. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:47.968961 14544 leader_election.cc:343] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:47.969522 14544 leader_election.cc:304] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:47.970180 15646 raft_consensus.cc:2749] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:47.976766 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_9776c6b19dd141b9ac63ce2c700c2a0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:47.978217 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9776c6b19dd141b9ac63ce2c700c2a0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:47.725067 14714 raft_consensus.cc:359] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.946925 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6d0fef50524476dbd47fc70e4bda319" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:47.984580 14542 leader_election.cc:343] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:47.985229 14714 raft_consensus.cc:385] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:47.985642 14714 raft_consensus.cc:740] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:47.955606 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a2dcb9ac62e4aac885d638f4e2b072e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:47.986601 14714 consensus_queue.cc:260] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:47.988925 14714 ts_tablet_manager.cc:1434] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.266s user 0.005s sys 0.000s
W20251211 08:57:48.012372 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7c04583bef74d84a49cdcd8a1305a88 consensus state has no leader
I20251211 08:57:47.971009 15299 rpcz_store.cc:276] 1211 08:57:45.576663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.576752 (+ 89us) service_pool.cc:225] Handling call
1211 08:57:46.743044 (+1166292us) ts_tablet_manager.cc:1151] Deleting tablet 7b90b0c944174b2a82696e3fee806934
1211 08:57:46.743078 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:47.361113 (+618035us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.633879 (+272766us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:47.865930 (+232051us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":157894,"tablet-delete.queue_time_us":1072599}
W20251211 08:57:48.029465 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 60570us
I20251211 08:57:48.030925 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 7b90b0c944174b2a82696e3fee806934 (table default.Table_e3e740f7ffee4b1683b5888496b1a384 [id=0f8c030be5504d859ef25176b6cbb4b1]) successfully deleted
I20251211 08:57:48.031539 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 375) took 2456 ms. Trace:
W20251211 08:57:48.035619 14544 leader_election.cc:343] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Not found: Tablet not found: 593df63696a04f85b00c301d93748e80
I20251211 08:57:48.037871 14544 leader_election.cc:304] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:48.036593 14714 tablet_bootstrap.cc:492] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:48.038712 15299 rpcz_store.cc:276] 1211 08:57:45.575097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.585771 (+ 10674us) service_pool.cc:225] Handling call
1211 08:57:48.031204 (+2445433us) ts_tablet_manager.cc:1151] Deleting tablet 3838fd220d034b3595f351b906188339
1211 08:57:48.031259 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.031473 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3637,"tablet-delete.queue_time_us":2345071}
I20251211 08:57:47.955037 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a2dcb9ac62e4aac885d638f4e2b072e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:47.975603 14722 raft_consensus.cc:359] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.042484 14722 raft_consensus.cc:385] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:48.042794 14722 raft_consensus.cc:740] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:48.043346 14722 consensus_queue.cc:260] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.037483 15030 tablet_bootstrap.cc:492] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:48.045223 14722 ts_tablet_manager.cc:1434] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.232s user 0.006s sys 0.000s
I20251211 08:57:48.055655 15646 raft_consensus.cc:2749] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:48.054967 14385 leader_election.cc:304] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:48.057109 15648 raft_consensus.cc:2804] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:48.057585 15648 raft_consensus.cc:697] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:48.058383 15648 consensus_queue.cc:237] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.063733 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 378) took 2161 ms. Trace:
I20251211 08:57:48.064041 15299 rpcz_store.cc:276] 1211 08:57:45.902483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.904765 (+ 2282us) service_pool.cc:225] Handling call
1211 08:57:48.063447 (+2158682us) ts_tablet_manager.cc:1151] Deleting tablet 6874fb018bc844b3aeeab7db8fc53e9a
1211 08:57:48.063483 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.063687 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2158001}
I20251211 08:57:48.065157 15299 tablet_replica.cc:333] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:48.065876 15299 raft_consensus.cc:2243] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251211 08:57:48.066046 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 3838fd220d034b3595f351b906188339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3838fd220d034b3595f351b906188339 already in progress: creating tablet
I20251211 08:57:48.066394 15299 raft_consensus.cc:2272] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:48.066638 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 6874fb018bc844b3aeeab7db8fc53e9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6874fb018bc844b3aeeab7db8fc53e9a already in progress: creating tablet
I20251211 08:57:48.068293 15299 ts_tablet_manager.cc:1916] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:48.081808 14544 leader_election.cc:343] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:48.082226 14544 leader_election.cc:304] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
W20251211 08:57:48.083242 14544 leader_election.cc:343] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:48.083612 14544 leader_election.cc:304] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:48.084277 15515 raft_consensus.cc:2749] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:48.084884 15515 raft_consensus.cc:2749] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:48.095321 14904 ts_tablet_manager.cc:1929] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:48.095695 14904 log.cc:1199] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/14fc87cb76024b5b8227dfe9939e1c89
I20251211 08:57:48.096439 14722 tablet_bootstrap.cc:492] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:48.112465 14722 tablet_bootstrap.cc:654] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.116901 15030 tablet_bootstrap.cc:654] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.116461 14714 tablet_bootstrap.cc:654] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.128824 15030 tablet_bootstrap.cc:492] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:48.129170 15030 ts_tablet_manager.cc:1403] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.189s user 0.005s sys 0.004s
I20251211 08:57:48.134199 15030 raft_consensus.cc:359] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.134822 15030 raft_consensus.cc:385] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:48.135051 15030 raft_consensus.cc:740] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:48.135605 15030 consensus_queue.cc:260] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.145119 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_873e2f0afae049eb85e3652c71880de5" } new_table_name: "default.Table_2d6ce3412a654c938bf9e15598110783" modify_external_catalogs: true
I20251211 08:57:48.156261 14714 tablet_bootstrap.cc:492] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:48.156656 14714 ts_tablet_manager.cc:1403] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.123s user 0.004s sys 0.006s
I20251211 08:57:48.158818 14714 raft_consensus.cc:359] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.159564 14714 raft_consensus.cc:385] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:48.159570 14905 ts_tablet_manager.cc:1929] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:48.159766 14714 raft_consensus.cc:740] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:48.159893 14905 log.cc:1199] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/98d14ba15dcd47f8b9e8035b3c7d477c
I20251211 08:57:48.168287 14905 ts_tablet_manager.cc:1950] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:48.168680 14722 tablet_bootstrap.cc:492] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:48.169154 14722 ts_tablet_manager.cc:1403] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.004s
I20251211 08:57:48.170795 15030 ts_tablet_manager.cc:1434] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20251211 08:57:48.173553 14714 consensus_queue.cc:260] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.175727 14714 ts_tablet_manager.cc:1434] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20251211 08:57:48.176798 14714 tablet_bootstrap.cc:492] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:48.096480 14904 ts_tablet_manager.cc:1950] T 14fc87cb76024b5b8227dfe9939e1c89 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:48.190750 14005 master_service.cc:954] ReplaceTablet: received request to replace tablet f3cf2b4d14ba45d79e52b6fdb0497d58 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:48.199954 15030 tablet_bootstrap.cc:492] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:48.205693 15030 tablet_bootstrap.cc:654] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.206137 14722 raft_consensus.cc:359] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.207399 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 295) took 5426 ms. Trace:
I20251211 08:57:48.207687 14905 rpcz_store.cc:276] 1211 08:57:42.780955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.781060 (+ 105us) service_pool.cc:225] Handling call
1211 08:57:47.958294 (+5177234us) ts_tablet_manager.cc:1151] Deleting tablet 98d14ba15dcd47f8b9e8035b3c7d477c
1211 08:57:47.958329 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.099488 (+141159us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.159368 (+ 59880us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.207345 (+ 47977us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5176795}
I20251211 08:57:48.208750 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 98d14ba15dcd47f8b9e8035b3c7d477c (table default.Table_a4bc9b5633444f8587bf740793fd55a2 [id=1a961233d708456bb4cf8d3fc39dc705]) successfully deleted
I20251211 08:57:48.210392 14905 tablet_replica.cc:333] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:48.211254 14905 raft_consensus.cc:2243] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Raft consensus shutting down.
I20251211 08:57:48.215492 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f33eb1889a164520a189fdca141a29c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:48.216646 14010 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a2e57210dc9747d2926a04ef6cdeb10f" } new_table_name: "default.Table_d9c25365b4374da1aab0e61d5148e89f" modify_external_catalogs: true
W20251211 08:57:48.216956 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f33eb1889a164520a189fdca141a29c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:48.219468 14012 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7919ce70cfc5400bb4f2515164c6b7a1" } modify_external_catalogs: true
I20251211 08:57:48.219897 14012 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7919ce70cfc5400bb4f2515164c6b7a1" } modify_external_catalogs: true
I20251211 08:57:48.225471 14722 raft_consensus.cc:385] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:48.225734 14722 raft_consensus.cc:740] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:48.229374 14276 tablet_service.cc:1505] Processing CreateTablet for tablet b4c19762a09c4adab6b7934da064b25b (DEFAULT_TABLE table=default.Table_759bc12a73cc4cab9c90f127a2522939 [id=86663d26fd884682a6731ddf9829f240]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.231760 14428 tablet_service.cc:1505] Processing CreateTablet for tablet b4c19762a09c4adab6b7934da064b25b (DEFAULT_TABLE table=default.Table_759bc12a73cc4cab9c90f127a2522939 [id=86663d26fd884682a6731ddf9829f240]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.232205 15648 raft_consensus.cc:493] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:48.232684 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c19762a09c4adab6b7934da064b25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.232071 14007 catalog_manager.cc:5654] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:48.232561 15648 raft_consensus.cc:515] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.233145 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_49a263a32c9940a5b61d8a430ebb61a7 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 3 got 0
I20251211 08:57:48.235473 14265 tablet_service.cc:1505] Processing CreateTablet for tablet 605ccab400ed4557bfd7c61cba4448e2 (DEFAULT_TABLE table=default.Table_481bef424de1464b95f1e738b5986299 [id=0eefe0f7c94a464a8914d3f35748b611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.240216 14418 tablet_service.cc:1505] Processing CreateTablet for tablet 605ccab400ed4557bfd7c61cba4448e2 (DEFAULT_TABLE table=default.Table_481bef424de1464b95f1e738b5986299 [id=0eefe0f7c94a464a8914d3f35748b611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.241253 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 605ccab400ed4557bfd7c61cba4448e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.243902 15648 leader_election.cc:290] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:48.244354 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 29869ac852e64489973feb8b0d86ab57 (DEFAULT_TABLE table=default.Table_b7ff902a532d4e7697811e15bcafde66 [id=5229dbf72eed45afb48ea77126334294]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.246877 14423 tablet_service.cc:1505] Processing CreateTablet for tablet 29869ac852e64489973feb8b0d86ab57 (DEFAULT_TABLE table=default.Table_b7ff902a532d4e7697811e15bcafde66 [id=5229dbf72eed45afb48ea77126334294]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.247728 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29869ac852e64489973feb8b0d86ab57. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:48.248428 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58d5ded96b9149f9ab0d9ea70e376af1 consensus state has no leader
W20251211 08:57:34.223299 15138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:57:48.250663 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 0d6339a0e795499a935b6bf106b5590b (DEFAULT_TABLE table=default.Table_d0cc7be666b34403b1626f0999117efe [id=b238e88cc87e493cb4b4f4701828a9f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.253566 14416 tablet_service.cc:1505] Processing CreateTablet for tablet 0d6339a0e795499a935b6bf106b5590b (DEFAULT_TABLE table=default.Table_d0cc7be666b34403b1626f0999117efe [id=b238e88cc87e493cb4b4f4701828a9f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.254590 14416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d6339a0e795499a935b6bf106b5590b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:48.214942 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 129959us
I20251211 08:57:48.233155 14578 tablet_service.cc:1505] Processing CreateTablet for tablet b4c19762a09c4adab6b7934da064b25b (DEFAULT_TABLE table=default.Table_759bc12a73cc4cab9c90f127a2522939 [id=86663d26fd884682a6731ddf9829f240]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.255403 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c19762a09c4adab6b7934da064b25b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:34.301754 15141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:57:48.243391 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 605ccab400ed4557bfd7c61cba4448e2 (DEFAULT_TABLE table=default.Table_481bef424de1464b95f1e738b5986299 [id=0eefe0f7c94a464a8914d3f35748b611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.260108 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 605ccab400ed4557bfd7c61cba4448e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.260308 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_e3591ad0b25d41499ce64dc2ea8ff7b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:48.261907 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3591ad0b25d41499ce64dc2ea8ff7b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:48.250363 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 29869ac852e64489973feb8b0d86ab57 (DEFAULT_TABLE table=default.Table_b7ff902a532d4e7697811e15bcafde66 [id=5229dbf72eed45afb48ea77126334294]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.263432 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29869ac852e64489973feb8b0d86ab57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.254359 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 0d6339a0e795499a935b6bf106b5590b (DEFAULT_TABLE table=default.Table_d0cc7be666b34403b1626f0999117efe [id=b238e88cc87e493cb4b4f4701828a9f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.267343 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d6339a0e795499a935b6bf106b5590b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:48.267736 15139 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 14063 milliseconds
I20251211 08:57:48.269183 14931 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251211 08:57:48.270013 14714 tablet_bootstrap.cc:654] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.270892 15648 raft_consensus.cc:493] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:48.271342 15648 raft_consensus.cc:515] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.273396 15648 leader_election.cc:290] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:48.275187 15515 raft_consensus.cc:493] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:48.275594 15515 raft_consensus.cc:515] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.277251 15515 leader_election.cc:290] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:48.279311 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "552de4c6d030418181f0400714d9ec31" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:48.280519 14542 leader_election.cc:343] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:48.270684 14931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:57:48.286840 15612 raft_consensus.cc:493] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:48.287232 15612 raft_consensus.cc:515] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.288883 15612 leader_election.cc:290] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:48.289731 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f938359354b545cfbac560b7d7888b9f" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:48.309257 14544 consensus_peers.cc:597] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:48.310201 14544 leader_election.cc:343] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:48.310524 14544 leader_election.cc:304] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:48.315068 14722 consensus_queue.cc:260] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:48.317806 14544 raft_consensus.cc:2686] T 98d14ba15dcd47f8b9e8035b3c7d477c P c5e624bcfd154cd586ef1dedb9940063: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251211 08:57:48.324124 14544 leader_election.cc:343] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Not found: Tablet not found: f909d82e0136470a810e1063d6769868
I20251211 08:57:48.327600 15299 ts_tablet_manager.cc:1929] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:48.330117 14722 ts_tablet_manager.cc:1434] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.161s user 0.003s sys 0.003s
I20251211 08:57:48.330644 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45c7d841bd3643a799a8b96e82d6d675" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:48.331393 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 343) took 2173 ms. Trace:
I20251211 08:57:48.330662 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da57b72207414512b9744b283bc869fe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:48.331743 14904 rpcz_store.cc:276] 1211 08:57:46.158197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.158310 (+ 113us) service_pool.cc:225] Handling call
1211 08:57:47.647834 (+1489524us) ts_tablet_manager.cc:1151] Deleting tablet 14fc87cb76024b5b8227dfe9939e1c89
1211 08:57:47.647882 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:47.819540 (+171658us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.095082 (+275542us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.331325 (+236243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1489010}
I20251211 08:57:48.332996 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 349) took 1948 ms. Trace:
I20251211 08:57:48.333276 14904 rpcz_store.cc:276] 1211 08:57:46.384834 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.402493 (+ 17659us) service_pool.cc:225] Handling call
1211 08:57:48.332732 (+1930239us) ts_tablet_manager.cc:1151] Deleting tablet 00e5c4bf71b44abca66b0e94995c533e
1211 08:57:48.332766 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.332954 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1929691}
I20251211 08:57:48.336663 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 14fc87cb76024b5b8227dfe9939e1c89 (table default.Table_fd2daf1c25534c25b34cf363453b0e67 [id=6e2d344612214ee382863680435345b7]) successfully deleted
W20251211 08:57:48.337272 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 00e5c4bf71b44abca66b0e94995c533e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00e5c4bf71b44abca66b0e94995c533e already in progress: creating tablet
I20251211 08:57:48.338892 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 348) took 1954 ms. Trace:
I20251211 08:57:48.339957 14904 rpcz_store.cc:276] 1211 08:57:46.384545 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.403118 (+ 18573us) service_pool.cc:225] Handling call
1211 08:57:48.334896 (+1931778us) ts_tablet_manager.cc:1151] Deleting tablet c6d419632b2a49e1975d4baa8ac85322
1211 08:57:48.337891 (+ 2995us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.338690 (+ 799us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1931188}
W20251211 08:57:48.342597 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet c6d419632b2a49e1975d4baa8ac85322 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6d419632b2a49e1975d4baa8ac85322 already in progress: creating tablet
I20251211 08:57:48.344736 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 355) took 1737 ms. Trace:
I20251211 08:57:48.346413 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d6339a0e795499a935b6bf106b5590b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:48.363744 14010 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47e3c0a8e8344e3aa688d3d6a48c15ed consensus state has no leader
I20251211 08:57:48.363444 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a15d62e9febf43b3a22ef81fb3feafb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:48.364410 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet f3cf2b4d14ba45d79e52b6fdb0497d58
I20251211 08:57:48.364555 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_05b7dc5c661e4002b1fee6521138d5a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:48.364843 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a15d62e9febf43b3a22ef81fb3feafb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:48.365705 14005 catalog_manager.cc:6724] ReplaceTablet: tablet f3cf2b4d14ba45d79e52b6fdb0497d58 deleted and replaced by tablet 4849a22585f24e388e3d8d6f4277c997
W20251211 08:57:48.365885 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05b7dc5c661e4002b1fee6521138d5a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:48.369665 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet f3cf2b4d14ba45d79e52b6fdb0497d58 with delete_type TABLET_DATA_DELETED (replaced by tablet 4849a22585f24e388e3d8d6f4277c997) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:48.369791 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c19762a09c4adab6b7934da064b25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.370231 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 605ccab400ed4557bfd7c61cba4448e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.370735 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29869ac852e64489973feb8b0d86ab57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.418110 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet aadc3c1ecd974ae3a0d491bd209047e6
I20251211 08:57:48.424288 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
I20251211 08:57:48.425113 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet c6d0fef50524476dbd47fc70e4bda319
I20251211 08:57:48.425199 14008 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_bf148c361a2346e88e4c9e4430b1abca [id=26c0a9d9b1f84372b18a54f045eb768c] alter complete (version 4)
W20251211 08:57:48.427835 14008 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 54) took 1550 ms (client timeout 1000 ms). Trace:
I20251211 08:57:48.430919 14011 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_b58bca5097ae4e3cab8c112bc4d17bbb [id=52ae4637827d424eb4d37362bf21cea2] alter complete (version 4)
W20251211 08:57:48.431370 14011 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 49) took 1461 ms (client timeout 1000 ms). Trace:
I20251211 08:57:48.345696 14904 rpcz_store.cc:276] 1211 08:57:46.606948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.607909 (+ 961us) service_pool.cc:225] Handling call
1211 08:57:48.343588 (+1735679us) ts_tablet_manager.cc:1151] Deleting tablet faa96580439242948a66bbf00aeb836a
1211 08:57:48.343812 (+ 224us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.344587 (+ 775us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1735091}
I20251211 08:57:48.433641 14904 tablet_replica.cc:333] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b: stopping tablet replica
W20251211 08:57:48.431608 14011 rpcz_store.cc:269] 1211 08:57:46.970170 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.970257 (+ 87us) service_pool.cc:225] Handling call
1211 08:57:47.054321 (+ 84064us) spinlock_profiling.cc:243] Waited 82.7 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:48.431313 (+1376992us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:47.608720 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:47.609031 (+ 311us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:47.609048 (+ 17us) write_op.cc:435] Acquired schema lock
1211 08:57:47.609057 (+ 9us) tablet.cc:660] Decoding operations
1211 08:57:47.609246 (+ 189us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:47.609276 (+ 30us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:47.609294 (+ 18us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:47.609417 (+ 123us) tablet.cc:699] Row locks acquired
1211 08:57:47.609428 (+ 11us) write_op.cc:260] PREPARE: finished
1211 08:57:47.609514 (+ 86us) write_op.cc:270] Start()
1211 08:57:47.609579 (+ 65us) write_op.cc:276] Timestamp: P: 1765443467609680 usec, L: 0
1211 08:57:47.609593 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:47.609949 (+ 356us) log.cc:844] Serialized 356 byte log entry
1211 08:57:47.617364 (+ 7415us) op_driver.cc:464] REPLICATION: finished
1211 08:57:47.617663 (+ 299us) write_op.cc:301] APPLY: starting
1211 08:57:47.617711 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:47.617780 (+ 69us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:47.617869 (+ 89us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:47.618006 (+ 137us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:47.618114 (+ 108us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:47.618151 (+ 37us) write_op.cc:312] APPLY: finished
1211 08:57:47.618283 (+ 132us) log.cc:844] Serialized 23 byte log entry
1211 08:57:47.618407 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:47.618457 (+ 50us) write_op.cc:454] Released schema lock
1211 08:57:47.618482 (+ 25us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:47.623790 (+ 5308us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:47.623808 (+ 18us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":190104320,"child_traces":[["op",{"apply.queue_time_us":179,"apply.run_cpu_time_us":992,"apply.run_wall_time_us":5052,"num_ops":1,"prepare.queue_time_us":508909,"prepare.run_cpu_time_us":1459,"prepare.run_wall_time_us":15064,"raft.queue_time_us":1838,"raft.run_cpu_time_us":486,"raft.run_wall_time_us":1882,"replication_time_us":7698}]]}
W20251211 08:57:48.444655 14004 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 79) took 971 ms (client timeout 1000 ms). Trace:
I20251211 08:57:48.451025 14722 tablet_bootstrap.cc:492] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:48.453590 14004 rpcz_store.cc:269] 1211 08:57:47.472890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.487419 (+ 14529us) service_pool.cc:225] Handling call
1211 08:57:48.444608 (+957189us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3}
I20251211 08:57:48.454278 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet aadc3c1ecd974ae3a0d491bd209047e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:48.455853 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet c6d0fef50524476dbd47fc70e4bda319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:48 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:48.325109 14905 pending_rounds.cc:70] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:57:48.367493 14576 tablet_service.cc:1552] Processing DeleteTablet for tablet f3cf2b4d14ba45d79e52b6fdb0497d58 with delete_type TABLET_DATA_DELETED (replaced by tablet 4849a22585f24e388e3d8d6f4277c997) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:48.457293 14905 pending_rounds.cc:77] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7231256435639001088 op_type: NO_OP noop_request { }
I20251211 08:57:48.457585 14905 raft_consensus.cc:2889] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:48.457851 14905 raft_consensus.cc:2272] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:48.421087 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet aadc3c1ecd974ae3a0d491bd209047e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:48.445531 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet c6d0fef50524476dbd47fc70e4bda319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:48 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:48.446228 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:36 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:48.459515 14905 ts_tablet_manager.cc:1916] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:48.463630 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet faa96580439242948a66bbf00aeb836a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faa96580439242948a66bbf00aeb836a already in progress: creating tablet
W20251211 08:57:48.428085 14008 rpcz_store.cc:269] 1211 08:57:46.877637 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.877736 (+ 99us) service_pool.cc:225] Handling call
1211 08:57:48.427758 (+1550022us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:47.430589 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:47.430963 (+ 374us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:47.430985 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:57:47.430996 (+ 11us) tablet.cc:660] Decoding operations
1211 08:57:47.431206 (+ 210us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:47.431227 (+ 21us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:47.431239 (+ 12us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:47.431325 (+ 86us) tablet.cc:699] Row locks acquired
1211 08:57:47.431331 (+ 6us) write_op.cc:260] PREPARE: finished
1211 08:57:47.431397 (+ 66us) write_op.cc:270] Start()
1211 08:57:47.431442 (+ 45us) write_op.cc:276] Timestamp: P: 1765443467431495 usec, L: 0
1211 08:57:47.431450 (+ 8us) op_driver.cc:348] REPLICATION: starting
1211 08:57:47.481217 (+ 49767us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b5400020920. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbef5a1 00007fc20bc03550 00007fc20bc8c7c2 00007fc20bc81be7 00007fc20bc81a24 00007fc20c0f34d1 00007fc20c0f2ca8 00007fc20c0f7941 00007fc20c0f78f9 00007fc20c0f7889 00007fc20c0f7851
1211 08:57:47.481661 (+ 444us) log.cc:844] Serialized 530 byte log entry
1211 08:57:47.511949 (+ 30288us) op_driver.cc:464] REPLICATION: finished
1211 08:57:47.512156 (+ 207us) write_op.cc:301] APPLY: starting
1211 08:57:47.512202 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:47.512279 (+ 77us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:47.512296 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:47.512438 (+ 142us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:47.512586 (+ 148us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:47.512628 (+ 42us) write_op.cc:312] APPLY: finished
1211 08:57:47.513334 (+ 706us) log.cc:844] Serialized 23 byte log entry
1211 08:57:47.513476 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:47.513554 (+ 78us) write_op.cc:454] Released schema lock
1211 08:57:47.513584 (+ 30us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:47.524200 (+ 10616us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbef5a1 00007fc20bc81aa8 00007fc20c0f34d1 00007fc20c0f2ca8 00007fc20c0f7941 00007fc20c0f78f9 00007fc20c0f7889 00007fc20c0f7851 00007fc20c0f6a7d 00000000005aa2b4 00000000005aa0e9
1211 08:57:47.524772 (+ 572us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:47.524802 (+ 30us) write_op.cc:454] Released schema lock
Related trace 'op':
1211 08:57:47.739106 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:47.739447 (+ 341us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:47.739466 (+ 19us) write_op.cc:435] Acquired schema lock
1211 08:57:47.739476 (+ 10us) tablet.cc:660] Decoding operations
1211 08:57:47.739741 (+ 265us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:47.739775 (+ 34us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:47.739793 (+ 18us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:47.739912 (+ 119us) tablet.cc:699] Row locks acquired
1211 08:57:47.739926 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:57:47.740013 (+ 87us) write_op.cc:270] Start()
1211 08:57:47.740083 (+ 70us) write_op.cc:276] Timestamp: P: 1765443467740018 usec, L: 0
1211 08:57:47.740095 (+ 12us) op_driver.cc:348] REPLICATION: starting
1211 08:57:47.740491 (+ 396us) log.cc:844] Serialized 356 byte log entry
1211 08:57:47.749515 (+ 9024us) op_driver.cc:464] REPLICATION: finished
1211 08:57:47.749793 (+ 278us) write_op.cc:301] APPLY: starting
1211 08:57:47.749837 (+ 44us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:47.749912 (+ 75us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:47.749980 (+ 68us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:47.750123 (+ 143us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:47.750227 (+ 104us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:47.750261 (+ 34us) write_op.cc:312] APPLY: finished
1211 08:57:47.750407 (+ 146us) log.cc:844] Serialized 23 byte log entry
1211 08:57:47.750538 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:47.750601 (+ 63us) write_op.cc:454] Released schema lock
1211 08:57:47.750626 (+ 25us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:47.758159 (+ 7533us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:47.758191 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":79,"apply.run_cpu_time_us":1607,"apply.run_wall_time_us":1606,"num_ops":1,"prepare.queue_time_us":530132,"prepare.run_cpu_time_us":2978,"prepare.run_wall_time_us":94161,"raft.queue_time_us":69,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":205,"replication_time_us":80420,"spinlock_wait_cycles":112557696}],["op",{"apply.queue_time_us":168,"apply.run_cpu_time_us":993,"apply.run_wall_time_us":1100,"num_ops":1,"prepare.queue_time_us":221651,"prepare.run_cpu_time_us":1661,"prepare.run_wall_time_us":19024,"raft.queue_time_us":217,"raft.run_cpu_time_us":524,"raft.run_wall_time_us":1127,"replication_time_us":9339}]]}
I20251211 08:57:48.479707 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8855dc73d29241daa7d7c423bcae474b" } new_table_name: "default.Table_289d1c8ae20f4dd893b92590045554ad" modify_external_catalogs: true
I20251211 08:57:48.483208 14904 raft_consensus.cc:2243] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:48.483635 14904 raft_consensus.cc:2272] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:48.372802 14714 tablet_bootstrap.cc:492] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:48.485340 14714 ts_tablet_manager.cc:1403] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.309s user 0.020s sys 0.012s
I20251211 08:57:48.487974 14007 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
W20251211 08:57:48.336207 14385 leader_election.cc:343] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:48.489019 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 152905us
W20251211 08:57:48.489552 14385 leader_election.cc:343] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:48.489324 14722 tablet_bootstrap.cc:654] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.496366 14714 raft_consensus.cc:359] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.497303 14714 raft_consensus.cc:385] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:48.497655 14714 raft_consensus.cc:740] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:48.506896 14714 consensus_queue.cc:260] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:48.508314 14004 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65a6feed325f4391a15553e8be914d2d consensus state has no leader
I20251211 08:57:48.515676 14416 tablet_service.cc:1552] Processing DeleteTablet for tablet f3cf2b4d14ba45d79e52b6fdb0497d58 with delete_type TABLET_DATA_DELETED (replaced by tablet 4849a22585f24e388e3d8d6f4277c997) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:48.516350 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet aadc3c1ecd974ae3a0d491bd209047e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:48.516858 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet c6d0fef50524476dbd47fc70e4bda319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:48 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:48.517220 14714 ts_tablet_manager.cc:1434] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20251211 08:57:48.518998 14931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:57:48.521149 14714 tablet_bootstrap.cc:492] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:48.532853 14004 catalog_manager.cc:5654] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:48.456874 14544 leader_election.cc:304] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:48.534303 14004 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
I20251211 08:57:48.534601 15670 raft_consensus.cc:2749] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:48.535328 14004 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 45c7d841bd3643a799a8b96e82d6d675 (table default.Table_f1453eb82ce2412c8208000d8ba7f240 [id=f08b1a30da03451ba864cb8f8e6d199b]). Expected version 1 got 0
W20251211 08:57:48.535447 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 211451us
W20251211 08:57:48.535982 14544 leader_election.cc:343] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Not found: Tablet not found: 4eec7694087145d8a49dd93904f95980
I20251211 08:57:48.536334 14544 leader_election.cc:304] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:48.536904 15670 raft_consensus.cc:2749] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:48.534806 15030 tablet_bootstrap.cc:492] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:48.539062 15030 ts_tablet_manager.cc:1403] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.339s user 0.011s sys 0.000s
I20251211 08:57:48.543540 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f32a1ccedd4f15bf8c79bc3f16e6bb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:48.543735 14722 tablet_bootstrap.cc:492] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:48.543996 14607 raft_consensus.cc:3060] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:48.544085 14722 ts_tablet_manager.cc:1403] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.093s user 0.006s sys 0.002s
I20251211 08:57:48.546960 14722 raft_consensus.cc:359] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.547531 14722 raft_consensus.cc:385] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:48.547752 14722 raft_consensus.cc:740] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:48.550349 14607 raft_consensus.cc:2468] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:48.550927 14931 hybrid_clock.cc:648] HybridClock initialized: now 1765443468550873 us; error 73 us; skew 500 ppm
I20251211 08:57:48.552083 14931 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:57:48.541209 15030 raft_consensus.cc:359] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.553290 15030 raft_consensus.cc:385] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:48.553522 15030 raft_consensus.cc:740] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:48.555521 15030 consensus_queue.cc:260] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.552743 14722 consensus_queue.cc:260] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.557372 14266 tablet_service.cc:1505] Processing CreateTablet for tablet eccc727462b447489372da09ea828ac2 (DEFAULT_TABLE table=default.Table_633182cd246844aba2af953c29c5d9e2 [id=04eb7857f3924c79b07b5d2670766644]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.558283 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eccc727462b447489372da09ea828ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.561688 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45c7d841bd3643a799a8b96e82d6d675" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:48.562153 14608 raft_consensus.cc:2468] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:48.564396 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da57b72207414512b9744b283bc869fe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:48.564805 14608 raft_consensus.cc:2468] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:48.568025 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6d0fef50524476dbd47fc70e4bda319" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:48.571269 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_2ff00d533f5a4fb58be17cfd01686207" } new_table_name: "default.Table_4a99523c26ff417bbf24b0deb1b04b09" modify_external_catalogs: true
I20251211 08:57:48.574364 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_bbc9ffbf679e4968b71046d57d9aae32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:48.575927 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbc9ffbf679e4968b71046d57d9aae32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:48.585433 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "552de4c6d030418181f0400714d9ec31" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:48.588604 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:39 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:48.592343 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fac41e1f10446d884cac85c4044167b consensus state has no leader
I20251211 08:57:48.594252 15299 log.cc:1199] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/14fc87cb76024b5b8227dfe9939e1c89
I20251211 08:57:48.595099 15299 ts_tablet_manager.cc:1950] T 14fc87cb76024b5b8227dfe9939e1c89 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:48.598065 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 379) took 2439 ms. Trace:
I20251211 08:57:48.598631 15030 ts_tablet_manager.cc:1434] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.059s user 0.006s sys 0.000s
I20251211 08:57:48.598893 15299 rpcz_store.cc:276] 1211 08:57:46.158742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.162840 (+ 4098us) service_pool.cc:225] Handling call
1211 08:57:48.065033 (+1902193us) ts_tablet_manager.cc:1151] Deleting tablet 14fc87cb76024b5b8227dfe9939e1c89
1211 08:57:48.065066 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.156401 (+ 91335us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.327397 (+170996us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.598012 (+270615us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1901671}
I20251211 08:57:48.599577 14722 ts_tablet_manager.cc:1434] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.055s user 0.003s sys 0.003s
W20251211 08:57:48.601334 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 65a6feed325f4391a15553e8be914d2d (table default.Table_289d1c8ae20f4dd893b92590045554ad [id=43bba7dab98342bc947c1c6c8f76db08], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 65a6feed325f4391a15553e8be914d2d consensus state has no leader
I20251211 08:57:48.601814 14904 ts_tablet_manager.cc:1916] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:48.616596 14722 tablet_bootstrap.cc:492] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:48.621182 14722 tablet_bootstrap.cc:654] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.623894 14009 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_cd9ec56dc08c48779e88a778d2917bb0" } modify_external_catalogs: true
I20251211 08:57:48.624343 15514 raft_consensus.cc:493] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:48.624486 14009 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_cd9ec56dc08c48779e88a778d2917bb0" } modify_external_catalogs: true
I20251211 08:57:48.624720 15514 raft_consensus.cc:515] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.626502 15514 leader_election.cc:290] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:48.614101 15030 tablet_bootstrap.cc:492] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:48.634042 15030 tablet_bootstrap.cc:654] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.641577 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd14de369e084828afb8a924c46c649b" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:48.649828 14271 tablet_service.cc:1505] Processing CreateTablet for tablet c3b539ac7e2c4b2f80407e01f4fb3364 (DEFAULT_TABLE table=default.Table_7ebe2394d4924d5b9e1345f599ad502d [id=e7e676e4259c4e2285e79b27962062ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.624094 14905 ts_tablet_manager.cc:1929] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251211 08:57:48.650780 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b539ac7e2c4b2f80407e01f4fb3364. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.639031 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f938359354b545cfbac560b7d7888b9f" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:48.650947 14905 log.cc:1199] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/2016208a33dc4a5cadd7ced8e85ccc65
I20251211 08:57:48.651438 14608 raft_consensus.cc:2393] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:48.651614 14905 ts_tablet_manager.cc:1950] T 2016208a33dc4a5cadd7ced8e85ccc65 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:57:48.653194 14544 leader_election.cc:343] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:48.653554 14544 leader_election.cc:304] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:48.654071 15670 raft_consensus.cc:2749] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:48.654901 14544 leader_election.cc:343] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:48.655265 14544 leader_election.cc:304] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:48.657447 15670 raft_consensus.cc:2749] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:48.664372 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f33eb1889a164520a189fdca141a29c7" } modify_external_catalogs: true
I20251211 08:57:48.664855 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f33eb1889a164520a189fdca141a29c7" } modify_external_catalogs: true
I20251211 08:57:48.667186 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f526612840624a878073018b98efcd87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:48.668548 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f526612840624a878073018b98efcd87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:48.670908 14544 leader_election.cc:343] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:48.671386 14544 consensus_peers.cc:597] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:48.672173 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 299) took 5773 ms. Trace:
I20251211 08:57:48.672472 14905 rpcz_store.cc:276] 1211 08:57:42.898624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.898747 (+ 123us) service_pool.cc:225] Handling call
1211 08:57:48.210267 (+5311520us) ts_tablet_manager.cc:1151] Deleting tablet 2016208a33dc4a5cadd7ced8e85ccc65
1211 08:57:48.210302 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.526451 (+316149us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.623891 (+ 97440us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.672127 (+ 48236us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14506,"tablet-delete.queue_time_us":5277726}
I20251211 08:57:48.673435 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 2016208a33dc4a5cadd7ced8e85ccc65 (table default.Table_face70f737874a76a60d1f4ea105380c [id=f93e9bee7bdd443c9eb3d158f1e40d06]) successfully deleted
I20251211 08:57:48.674108 14905 tablet_replica.cc:333] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:48.674202 14230 leader_election.cc:304] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:48.674690 14905 raft_consensus.cc:2243] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:48.675113 14905 raft_consensus.cc:2272] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:48.675401 15612 raft_consensus.cc:3060] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:48.676775 14905 ts_tablet_manager.cc:1916] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:48.684738 14722 tablet_bootstrap.cc:492] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:48.685207 14722 ts_tablet_manager.cc:1403] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.003s
I20251211 08:57:48.685824 14010 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 8ef12b71a94a414db3d5cec2c9bc4e41
I20251211 08:57:48.687394 14722 raft_consensus.cc:359] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.687945 14722 raft_consensus.cc:385] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:48.688222 14722 raft_consensus.cc:740] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:48.688756 14722 consensus_queue.cc:260] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.691501 15030 tablet_bootstrap.cc:492] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:48.691890 15030 ts_tablet_manager.cc:1403] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.078s user 0.008s sys 0.000s
I20251211 08:57:48.693980 15030 raft_consensus.cc:359] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.694522 15030 raft_consensus.cc:385] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:48.694772 15030 raft_consensus.cc:740] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:48.695482 15030 consensus_queue.cc:260] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.695675 14584 tablet_service.cc:1505] Processing CreateTablet for tablet eccc727462b447489372da09ea828ac2 (DEFAULT_TABLE table=default.Table_633182cd246844aba2af953c29c5d9e2 [id=04eb7857f3924c79b07b5d2670766644]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.697243 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eccc727462b447489372da09ea828ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.698693 15030 ts_tablet_manager.cc:1434] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251211 08:57:48.703087 15612 raft_consensus.cc:2749] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:48.706540 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2d8c6d6f376d4fe6909153c203750ac3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:48.709848 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f1c4e660ec174fa5b59704d5c6d8483c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:48.713016 14011 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f1a804079b754f0e84b85a175c773a3f" } new_table_name: "default.Table_c264c2b8de9a4f97b223eb3bfef06afb" modify_external_catalogs: true
I20251211 08:57:48.730448 14722 ts_tablet_manager.cc:1434] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.045s user 0.003s sys 0.002s
I20251211 08:57:48.731305 14722 tablet_bootstrap.cc:492] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:48.732827 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d8c6d6f376d4fe6909153c203750ac3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:48.738226 15030 tablet_bootstrap.cc:492] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
W20251211 08:57:48.739493 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1c4e660ec174fa5b59704d5c6d8483c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:48.749264 14722 tablet_bootstrap.cc:654] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.751538 15030 tablet_bootstrap.cc:654] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.756271 15612 consensus_queue.cc:1048] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:48.761817 14544 consensus_peers.cc:597] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14fc87cb76024b5b8227dfe9939e1c89. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:48.763425 14905 ts_tablet_manager.cc:1929] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:48.763727 14905 log.cc:1199] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/9ccfcacb5b294a628926d34cd31b137e
I20251211 08:57:48.771025 14905 ts_tablet_manager.cc:1950] T 9ccfcacb5b294a628926d34cd31b137e P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:48.772665 15030 tablet_bootstrap.cc:492] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:48.773059 15030 ts_tablet_manager.cc:1403] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.000s
I20251211 08:57:48.775364 15030 raft_consensus.cc:359] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.775998 15030 raft_consensus.cc:385] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:48.776278 15030 raft_consensus.cc:740] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:48.778261 14722 tablet_bootstrap.cc:492] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:48.778671 14722 ts_tablet_manager.cc:1403] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.000s
I20251211 08:57:48.780887 14722 raft_consensus.cc:359] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.781525 14722 raft_consensus.cc:385] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:48.781798 14722 raft_consensus.cc:740] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:48.784627 15612 raft_consensus.cc:493] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:48.785091 15612 raft_consensus.cc:515] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.788074 15612 leader_election.cc:290] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:48.788053 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0e0ee99989d41c6af82e330ab8e353c" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:48.795801 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_6b072b26d1564613a46376ccb5bb2635" } new_table_name: "default.Table_6c1a460a14a04dfb9cecebd86166d475" modify_external_catalogs: true
W20251211 08:57:48.803069 14931 net_util.cc:715] Time spent look up canonical hostname for localhost 'dist-test-slave-0dfs': real 0.249s user 0.004s sys 0.000s
I20251211 08:57:48.809602 15030 consensus_queue.cc:260] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.810320 15514 raft_consensus.cc:493] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:48.810674 15514 raft_consensus.cc:515] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.814527 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080d1700df4a4df09acda65c6d021d96" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:48.814554 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_e0dbfa29ddc84b3e87f44bc9ceb301a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:48.815882 14544 leader_election.cc:343] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:48.816143 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0dbfa29ddc84b3e87f44bc9ceb301a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:48.816700 15514 leader_election.cc:290] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:48.818053 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 300) took 5849 ms. Trace:
I20251211 08:57:48.818301 14905 rpcz_store.cc:276] 1211 08:57:42.968334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:42.968463 (+ 129us) service_pool.cc:225] Handling call
1211 08:57:48.673994 (+5705531us) ts_tablet_manager.cc:1151] Deleting tablet 9ccfcacb5b294a628926d34cd31b137e
1211 08:57:48.674026 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.713415 (+ 39389us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.763239 (+ 49824us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.818005 (+ 54766us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33794,"tablet-delete.queue_time_us":5704979}
I20251211 08:57:48.617767 14714 tablet_bootstrap.cc:654] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.819345 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 9ccfcacb5b294a628926d34cd31b137e (table default.Table_93954115dc5d4d2283faa424e83608b0 [id=cb30302a7db04098ae9a5a92ce055d40]) successfully deleted
I20251211 08:57:48.819782 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 14fc87cb76024b5b8227dfe9939e1c89 (table default.Table_fd2daf1c25534c25b34cf363453b0e67 [id=6e2d344612214ee382863680435345b7]) successfully deleted
I20251211 08:57:48.820838 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 384) took 2349 ms. Trace:
I20251211 08:57:48.623173 14382 leader_election.cc:304] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:48.627944 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd14de369e084828afb8a924c46c649b" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:48.645843 14428 tablet_service.cc:1505] Processing CreateTablet for tablet eccc727462b447489372da09ea828ac2 (DEFAULT_TABLE table=default.Table_633182cd246844aba2af953c29c5d9e2 [id=04eb7857f3924c79b07b5d2670766644]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.823033 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eccc727462b447489372da09ea828ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.824014 15030 ts_tablet_manager.cc:1434] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.051s user 0.007s sys 0.000s
I20251211 08:57:48.747582 15650 consensus_queue.cc:1048] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:48.827477 15650 raft_consensus.cc:2804] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:48.827777 15650 raft_consensus.cc:493] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:48.828083 15650 raft_consensus.cc:3060] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:48.828694 15030 tablet_bootstrap.cc:492] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:48.774123 15648 raft_consensus.cc:493] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:48.831009 15648 raft_consensus.cc:515] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.832700 15648 leader_election.cc:290] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:48.778931 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet da57b72207414512b9744b283bc869fe with delete_type TABLET_DATA_DELETED (replaced by tablet 27cb4228204c4176b1a348960efd5b26) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:48.788228 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0e0ee99989d41c6af82e330ab8e353c" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:48.811483 14416 tablet_service.cc:1505] Processing CreateTablet for tablet c3b539ac7e2c4b2f80407e01f4fb3364 (DEFAULT_TABLE table=default.Table_7ebe2394d4924d5b9e1345f599ad502d [id=e7e676e4259c4e2285e79b27962062ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.834439 14416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b539ac7e2c4b2f80407e01f4fb3364. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.834344 14231 leader_election.cc:304] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:48.834790 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 301) took 5807 ms. Trace:
I20251211 08:57:48.835302 15612 raft_consensus.cc:2749] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:48.835115 14905 rpcz_store.cc:276] 1211 08:57:43.027006 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.027134 (+ 128us) service_pool.cc:225] Handling call
1211 08:57:48.824454 (+5797320us) ts_tablet_manager.cc:1151] Deleting tablet 7af7c9ab27a64a1681f9874a922e986f
1211 08:57:48.829651 (+ 5197us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.834724 (+ 5073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5796753}
W20251211 08:57:48.836212 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 7af7c9ab27a64a1681f9874a922e986f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7af7c9ab27a64a1681f9874a922e986f already in progress: creating tablet
I20251211 08:57:48.836977 14905 tablet_replica.cc:333] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:48.837610 14905 raft_consensus.cc:2243] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:48.838065 14905 raft_consensus.cc:2272] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:48.839406 14542 leader_election.cc:343] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:48.839664 14905 ts_tablet_manager.cc:1916] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:48.839761 14542 leader_election.cc:304] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:48.817477 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080d1700df4a4df09acda65c6d021d96" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:48.841104 14011 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 680c8f54e75a4291bb4d769d94e21552 consensus state has no leader
W20251211 08:57:48.842603 14542 leader_election.cc:343] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:48.842509 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49015c5d4f2b4d07b547a9df57c992eb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:48.842968 14542 leader_election.cc:304] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:48.843773 15670 raft_consensus.cc:2749] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:48.844254 15670 raft_consensus.cc:2749] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:48.845504 14385 leader_election.cc:343] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:48.846529 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 223508us
I20251211 08:57:48.848812 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49015c5d4f2b4d07b547a9df57c992eb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:48.849448 14605 raft_consensus.cc:2468] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:48.852208 14382 leader_election.cc:304] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:48.852913 15696 raft_consensus.cc:2804] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:48.853240 15696 raft_consensus.cc:493] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:48.855154 15696 raft_consensus.cc:3060] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:48.821161 15299 rpcz_store.cc:276] 1211 08:57:46.471197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.471293 (+ 96us) service_pool.cc:225] Handling call
1211 08:57:48.820547 (+2349254us) ts_tablet_manager.cc:1151] Deleting tablet c6d419632b2a49e1975d4baa8ac85322
1211 08:57:48.820583 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.820794 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2348770}
I20251211 08:57:48.854040 15694 consensus_queue.cc:1048] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:48.863030 14382 leader_election.cc:304] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
W20251211 08:57:48.863353 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet c6d419632b2a49e1975d4baa8ac85322 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6d419632b2a49e1975d4baa8ac85322 already in progress: creating tablet
I20251211 08:57:48.863626 15697 raft_consensus.cc:2804] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:48.867610 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 677ef16a0610477b8ea05233a040cde4 (DEFAULT_TABLE table=default.Table_6fd146e5066b4fb9b651b986d7992aa8 [id=a0623dfc11d54c6e985a7115d6bc77fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.868590 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 677ef16a0610477b8ea05233a040cde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.867029 14586 tablet_service.cc:1505] Processing CreateTablet for tablet c3b539ac7e2c4b2f80407e01f4fb3364 (DEFAULT_TABLE table=default.Table_7ebe2394d4924d5b9e1345f599ad502d [id=e7e676e4259c4e2285e79b27962062ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.874707 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b539ac7e2c4b2f80407e01f4fb3364. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.802695 14256 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:46838 (request call id 59) took 1026 ms. Trace:
I20251211 08:57:48.886847 14256 rpcz_store.cc:276] 1211 08:57:47.776184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.776365 (+ 181us) service_pool.cc:225] Handling call
1211 08:57:48.802541 (+1026176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:57:48.892859 14722 consensus_queue.cc:260] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.858667 15030 tablet_bootstrap.cc:654] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:48.908524 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2d03f7f268c3461ca7e972ca69273fef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:48.910089 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d03f7f268c3461ca7e972ca69273fef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:48.916549 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 48575fa3f45349b9887bfc1cdfebc958 (DEFAULT_TABLE table=default.Table_e4cd2059809846e78020a7513c96df7e [id=1b3efd8519a04b13af86aa631f596f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.916872 14271 tablet_service.cc:1505] Processing CreateTablet for tablet bb39206fffbd4bba95c91432a95b95ff (DEFAULT_TABLE table=default.Table_b39d68e7f1b84e4b88959642577ec864 [id=9a0b33e6771f496eb5142be22b1cfe74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.917701 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48575fa3f45349b9887bfc1cdfebc958. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.917738 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb39206fffbd4bba95c91432a95b95ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.921576 14585 tablet_service.cc:1505] Processing CreateTablet for tablet bb39206fffbd4bba95c91432a95b95ff (DEFAULT_TABLE table=default.Table_b39d68e7f1b84e4b88959642577ec864 [id=9a0b33e6771f496eb5142be22b1cfe74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.922258 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 65a6feed325f4391a15553e8be914d2d (DEFAULT_TABLE table=default.Table_289d1c8ae20f4dd893b92590045554ad [id=43bba7dab98342bc947c1c6c8f76db08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.922502 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb39206fffbd4bba95c91432a95b95ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.923179 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a6feed325f4391a15553e8be914d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.950743 14905 ts_tablet_manager.cc:1929] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:48.951179 14905 log.cc:1199] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/4eec7694087145d8a49dd93904f95980
I20251211 08:57:48.951826 14905 ts_tablet_manager.cc:1950] T 4eec7694087145d8a49dd93904f95980 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:48.958782 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 302) took 5866 ms. Trace:
W20251211 08:57:48.959231 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e4c3a63cd7c4cc98a240e459e84a2ac consensus state has no leader
I20251211 08:57:48.959153 14905 rpcz_store.cc:276] 1211 08:57:43.092271 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.116043 (+ 23772us) service_pool.cc:225] Handling call
1211 08:57:48.836845 (+5720802us) ts_tablet_manager.cc:1151] Deleting tablet 4eec7694087145d8a49dd93904f95980
1211 08:57:48.836882 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.934195 (+ 97313us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.950511 (+ 16316us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.958723 (+ 8212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5716706}
I20251211 08:57:48.960455 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 4eec7694087145d8a49dd93904f95980 (table default.Table_0551d84bbd1b41399aab17ec5d23007c [id=b57216300ae943de97021180869ef1cb]) successfully deleted
I20251211 08:57:48.962180 14905 tablet_replica.cc:333] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:48.974851 14905 raft_consensus.cc:2243] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:48.983350 14905 raft_consensus.cc:2272] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:48.983974 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 65a6feed325f4391a15553e8be914d2d (DEFAULT_TABLE table=default.Table_289d1c8ae20f4dd893b92590045554ad [id=43bba7dab98342bc947c1c6c8f76db08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:48.984961 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a6feed325f4391a15553e8be914d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:48.985656 15612 raft_consensus.cc:493] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:48.986199 15612 raft_consensus.cc:515] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:48.988595 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c2fb2d9564c43ee84b8dfcea8ec54b0" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:48.989377 14608 raft_consensus.cc:2393] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:48.987987 14904 ts_tablet_manager.cc:1929] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:49.000013 14904 log.cc:1199] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/4e813a4c73334c839bfe6af2119e259c
I20251211 08:57:48.990520 14265 tablet_service.cc:1505] Processing CreateTablet for tablet 593a90d88b194bb5a1c37e0e7e9097ef (DEFAULT_TABLE table=default.Table_1eb40c19ca894f47ba26da25462c5c3a [id=62e4ec0c216b41eebee5490eaba6ef5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.000743 14904 ts_tablet_manager.cc:1950] T 4e813a4c73334c839bfe6af2119e259c P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:49.000805 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593a90d88b194bb5a1c37e0e7e9097ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.001425 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 356) took 2316 ms. Trace:
I20251211 08:57:49.001717 14904 rpcz_store.cc:276] 1211 08:57:46.684717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.694849 (+ 10132us) service_pool.cc:225] Handling call
1211 08:57:48.433511 (+1738662us) ts_tablet_manager.cc:1151] Deleting tablet 4e813a4c73334c839bfe6af2119e259c
1211 08:57:48.433545 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.876664 (+443119us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:48.987807 (+111143us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.001381 (+ 13574us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1738088}
I20251211 08:57:48.988734 15612 leader_election.cc:290] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:49.002806 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 4e813a4c73334c839bfe6af2119e259c (table default.Table_358b3598ddbe4cc2831ba06d3600ac09 [id=ffaa7c2a60924b0cbdc1254bfad98189]) successfully deleted
I20251211 08:57:49.020874 14266 tablet_service.cc:1505] Processing CreateTablet for tablet df7611ed7bf5459e9fc81d172ed0f57d (DEFAULT_TABLE table=default.Table_fd2daf1c25534c25b34cf363453b0e67 [id=6e2d344612214ee382863680435345b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.021822 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7611ed7bf5459e9fc81d172ed0f57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.023927 14905 ts_tablet_manager.cc:1916] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:49.026865 15514 raft_consensus.cc:493] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:49.002753 14006 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_49a263a32c9940a5b61d8a430ebb61a7 [id=69dd3ef114034ac7a3d60b8a0fde70b6]). Expected version 3 got 0
I20251211 08:57:49.027223 15514 raft_consensus.cc:515] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.028697 15514 leader_election.cc:290] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:49.031489 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 98d14ba15dcd47f8b9e8035b3c7d477c with delete_type TABLET_DATA_DELETED (replaced by tablet 8d39234b58734161b3bcfb67d0df1bd4) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:49.032153 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cbd18e927c843e7b0aee7e0ccb1b06d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:49.033211 14544 leader_election.cc:343] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:49.035303 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 593a90d88b194bb5a1c37e0e7e9097ef (DEFAULT_TABLE table=default.Table_1eb40c19ca894f47ba26da25462c5c3a [id=62e4ec0c216b41eebee5490eaba6ef5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.035965 15030 tablet_bootstrap.cc:492] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:49.036224 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593a90d88b194bb5a1c37e0e7e9097ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.036353 15030 ts_tablet_manager.cc:1403] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.208s user 0.007s sys 0.008s
I20251211 08:57:49.038605 15030 raft_consensus.cc:359] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.039191 15030 raft_consensus.cc:385] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:49.039430 15030 raft_consensus.cc:740] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:49.039955 15030 consensus_queue.cc:260] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.041821 15030 ts_tablet_manager.cc:1434] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251211 08:57:49.044687 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_6bd043cbc5024c4c9881e15f002ce3cf" } new_table_name: "default.Table_90bc933b06374a368b9df13947262958" modify_external_catalogs: true
I20251211 08:57:48.879448 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 677ef16a0610477b8ea05233a040cde4 (DEFAULT_TABLE table=default.Table_6fd146e5066b4fb9b651b986d7992aa8 [id=a0623dfc11d54c6e985a7115d6bc77fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.050623 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 677ef16a0610477b8ea05233a040cde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.052928 14904 tablet_replica.cc:333] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:49.054569 14008 master_service.cc:954] ReplaceTablet: received request to replace tablet f2f095b59ecf44ffabaa0ac22212938b from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:49.059667 14580 tablet_service.cc:1505] Processing CreateTablet for tablet df7611ed7bf5459e9fc81d172ed0f57d (DEFAULT_TABLE table=default.Table_fd2daf1c25534c25b34cf363453b0e67 [id=6e2d344612214ee382863680435345b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.061939 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7611ed7bf5459e9fc81d172ed0f57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.062572 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 48575fa3f45349b9887bfc1cdfebc958 (DEFAULT_TABLE table=default.Table_e4cd2059809846e78020a7513c96df7e [id=1b3efd8519a04b13af86aa631f596f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.063556 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48575fa3f45349b9887bfc1cdfebc958. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.064795 15030 tablet_bootstrap.cc:492] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:48.864436 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 385) took 2392 ms. Trace:
I20251211 08:57:48.889385 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 49015c5d4f2b4d07b547a9df57c992eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:49.066385 15299 rpcz_store.cc:276] 1211 08:57:46.472162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.472245 (+ 83us) service_pool.cc:225] Handling call
1211 08:57:48.864143 (+2391898us) ts_tablet_manager.cc:1151] Deleting tablet 00e5c4bf71b44abca66b0e94995c533e
1211 08:57:48.864181 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:48.864390 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2391487}
I20251211 08:57:48.866875 14418 tablet_service.cc:1505] Processing CreateTablet for tablet 677ef16a0610477b8ea05233a040cde4 (DEFAULT_TABLE table=default.Table_6fd146e5066b4fb9b651b986d7992aa8 [id=a0623dfc11d54c6e985a7115d6bc77fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.067744 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 677ef16a0610477b8ea05233a040cde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.068637 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:57:49.070977 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 4ab48f6288494005ad42208e2e97ddf2
I20251211 08:57:49.072086 14299 raft_consensus.cc:3060] T c0e0ee99989d41c6af82e330ab8e353c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:49.072707 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ab48f6288494005ad42208e2e97ddf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:48 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:49.073976 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_49a263a32c9940a5b61d8a430ebb61a7" } modify_external_catalogs: true
I20251211 08:57:49.074435 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_49a263a32c9940a5b61d8a430ebb61a7" } modify_external_catalogs: true
I20251211 08:57:49.079182 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ab48f6288494005ad42208e2e97ddf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:48 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:49.082252 14266 tablet_service.cc:1505] Processing CreateTablet for tablet a778ba38fc87459baa137671aec46e5c (DEFAULT_TABLE table=default.Table_ca924cbdb1e6438aa8cf80713dad7885 [id=f91c545b59504577908968374c8c9475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.083686 15030 tablet_bootstrap.cc:654] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:49.086478 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a778ba38fc87459baa137671aec46e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.092248 14580 tablet_service.cc:1505] Processing CreateTablet for tablet a778ba38fc87459baa137671aec46e5c (DEFAULT_TABLE table=default.Table_ca924cbdb1e6438aa8cf80713dad7885 [id=f91c545b59504577908968374c8c9475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.093197 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a778ba38fc87459baa137671aec46e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.105789 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet f2f095b59ecf44ffabaa0ac22212938b
I20251211 08:57:49.107717 14008 catalog_manager.cc:6724] ReplaceTablet: tablet f2f095b59ecf44ffabaa0ac22212938b deleted and replaced by tablet b3ad595efe5241e8a4f40b16be2ccd63
I20251211 08:57:49.120064 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet f2f095b59ecf44ffabaa0ac22212938b with delete_type TABLET_DATA_DELETED (replaced by tablet b3ad595efe5241e8a4f40b16be2ccd63) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:49.120764 14931 webserver.cc:492] Webserver started at http://127.13.146.62:43581/ using document root <none> and password file <none>
I20251211 08:57:49.122614 14931 fs_manager.cc:362] Metadata directory not provided
I20251211 08:57:49.123001 14931 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:57:49.090008 14905 ts_tablet_manager.cc:1929] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251211 08:57:49.115896 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet f2f095b59ecf44ffabaa0ac22212938b with delete_type TABLET_DATA_DELETED (replaced by tablet b3ad595efe5241e8a4f40b16be2ccd63) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:49.127082 14905 log.cc:1199] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/593df63696a04f85b00c301d93748e80
I20251211 08:57:49.119586 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet da57b72207414512b9744b283bc869fe with delete_type TABLET_DATA_DELETED (replaced by tablet 27cb4228204c4176b1a348960efd5b26) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:49.119191 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet e9905227c0a844d199362a442e7d630c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:49.128278 14905 ts_tablet_manager.cc:1950] T 593df63696a04f85b00c301d93748e80 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:49.129978 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 303) took 6012 ms. Trace:
I20251211 08:57:49.130273 14905 rpcz_store.cc:276] 1211 08:57:43.117541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.118527 (+ 986us) service_pool.cc:225] Handling call
1211 08:57:48.962049 (+5843522us) ts_tablet_manager.cc:1151] Deleting tablet 593df63696a04f85b00c301d93748e80
1211 08:57:48.962086 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.030911 (+ 68825us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.089813 (+ 58902us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.129930 (+ 40117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9730,"tablet-delete.queue_time_us":5841315}
I20251211 08:57:49.131466 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 593df63696a04f85b00c301d93748e80 (table default.Table_da883b3cac244bc0aee81f7c17ac81a6 [id=ab93470ba8f54070be718c240d49aa13]) successfully deleted
I20251211 08:57:49.132282 14905 tablet_replica.cc:333] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:49.133487 14905 raft_consensus.cc:2243] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251211 08:57:49.137349 14231 consensus_peers.cc:597] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5b462cf21e3420c8e323a4ac4f04abe. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:48.863968 15697 raft_consensus.cc:493] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:49.142700 14271 tablet_service.cc:1505] Processing CreateTablet for tablet 7b337f08cb0d4e78a611c03229252d54 (DEFAULT_TABLE table=default.Table_55467d689ec140f2a3ed64cc4ec0d806 [id=a1823ac371e54801bafcfdfc71d7901e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.144285 14714 tablet_bootstrap.cc:492] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:49.144670 14714 ts_tablet_manager.cc:1403] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.624s user 0.014s sys 0.001s
I20251211 08:57:49.144894 14722 ts_tablet_manager.cc:1434] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.366s user 0.019s sys 0.015s
I20251211 08:57:48.885242 15650 raft_consensus.cc:515] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.149518 14722 tablet_bootstrap.cc:492] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:49.142222 15697 raft_consensus.cc:3060] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:49.159350 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b337f08cb0d4e78a611c03229252d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.159684 14905 raft_consensus.cc:2272] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:49.159704 15650 leader_election.cc:290] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:49.160504 14265 tablet_service.cc:1505] Processing CreateTablet for tablet 7b6cdf1adae34ad2908233ce7323cf65 (DEFAULT_TABLE table=default.Table_9294d4662790426dbb71410c2c874bc0 [id=eec20447308f431092b2b3736ae03d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.161458 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b6cdf1adae34ad2908233ce7323cf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.161401 15696 raft_consensus.cc:515] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.156026 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 7b337f08cb0d4e78a611c03229252d54 (DEFAULT_TABLE table=default.Table_55467d689ec140f2a3ed64cc4ec0d806 [id=a1823ac371e54801bafcfdfc71d7901e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.158594 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 7b6cdf1adae34ad2908233ce7323cf65 (DEFAULT_TABLE table=default.Table_9294d4662790426dbb71410c2c874bc0 [id=eec20447308f431092b2b3736ae03d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.163129 15514 raft_consensus.cc:493] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:49.163359 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b6cdf1adae34ad2908233ce7323cf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.163480 15514 raft_consensus.cc:515] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.165132 15514 leader_election.cc:290] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:49.171339 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b337f08cb0d4e78a611c03229252d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.172150 14905 ts_tablet_manager.cc:1916] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:49.175421 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3567c7fe6104474f8d550cc0d2855ed6 consensus state has no leader
I20251211 08:57:49.174116 14714 raft_consensus.cc:359] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.175884 14714 raft_consensus.cc:385] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:49.176126 14714 raft_consensus.cc:740] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:49.176610 14714 consensus_queue.cc:260] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.179337 15696 leader_election.cc:290] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:49.184971 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_0c957f35d448462b9e819895f4832873"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:49.186615 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c957f35d448462b9e819895f4832873 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:49.187709 14904 raft_consensus.cc:2243] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:49.182579 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet e15483f539f24f1ba1b442663d66fd45
I20251211 08:57:49.188881 14714 ts_tablet_manager.cc:1434] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.044s user 0.004s sys 0.001s
I20251211 08:57:49.190646 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet e15483f539f24f1ba1b442663d66fd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:49.192682 14904 raft_consensus.cc:2272] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:49.195159 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cbd18e927c843e7b0aee7e0ccb1b06d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:49.195518 14904 ts_tablet_manager.cc:1916] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:49.196405 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9905227c0a844d199362a442e7d630c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:49.199242 14542 consensus_peers.cc:597] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14fc87cb76024b5b8227dfe9939e1c89. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:49.200050 14542 leader_election.cc:343] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:49.200212 14008 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_31fcfbed4a2c4e8aa05fb42c261bf1d5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 2 got 0
I20251211 08:57:49.200417 14542 leader_election.cc:304] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:49.201315 15514 raft_consensus.cc:2749] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:49.202162 14542 leader_election.cc:343] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:49.203769 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 00e5c4bf71b44abca66b0e94995c533e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00e5c4bf71b44abca66b0e94995c533e already in progress: creating tablet
I20251211 08:57:49.204144 14722 tablet_bootstrap.cc:654] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:49.204787 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45c7d841bd3643a799a8b96e82d6d675" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:49.205648 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da57b72207414512b9744b283bc869fe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:49.209317 14423 tablet_service.cc:1505] Processing CreateTablet for tablet bb39206fffbd4bba95c91432a95b95ff (DEFAULT_TABLE table=default.Table_b39d68e7f1b84e4b88959642577ec864 [id=9a0b33e6771f496eb5142be22b1cfe74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.210188 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb39206fffbd4bba95c91432a95b95ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.210400 14417 tablet_service.cc:1505] Processing CreateTablet for tablet 593a90d88b194bb5a1c37e0e7e9097ef (DEFAULT_TABLE table=default.Table_1eb40c19ca894f47ba26da25462c5c3a [id=62e4ec0c216b41eebee5490eaba6ef5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.211241 14417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593a90d88b194bb5a1c37e0e7e9097ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.211287 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 65a6feed325f4391a15553e8be914d2d (DEFAULT_TABLE table=default.Table_289d1c8ae20f4dd893b92590045554ad [id=43bba7dab98342bc947c1c6c8f76db08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.212275 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a6feed325f4391a15553e8be914d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.212572 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 9fac41e1f10446d884cac85c4044167b (DEFAULT_TABLE table=default.Table_4a99523c26ff417bbf24b0deb1b04b09 [id=61d1d967c6864029bac33248ffcbbced]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.213418 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fac41e1f10446d884cac85c4044167b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.215574 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ab48f6288494005ad42208e2e97ddf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:48 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:49.215708 14424 tablet_service.cc:1505] Processing CreateTablet for tablet df7611ed7bf5459e9fc81d172ed0f57d (DEFAULT_TABLE table=default.Table_fd2daf1c25534c25b34cf363453b0e67 [id=6e2d344612214ee382863680435345b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.216579 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7611ed7bf5459e9fc81d172ed0f57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.216531 14426 tablet_service.cc:1505] Processing CreateTablet for tablet a778ba38fc87459baa137671aec46e5c (DEFAULT_TABLE table=default.Table_ca924cbdb1e6438aa8cf80713dad7885 [id=f91c545b59504577908968374c8c9475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.216967 14416 tablet_service.cc:1505] Processing CreateTablet for tablet 48575fa3f45349b9887bfc1cdfebc958 (DEFAULT_TABLE table=default.Table_e4cd2059809846e78020a7513c96df7e [id=1b3efd8519a04b13af86aa631f596f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.217563 15686 raft_consensus.cc:493] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:49.218036 15686 raft_consensus.cc:515] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.218894 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a778ba38fc87459baa137671aec46e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.218950 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet f2f095b59ecf44ffabaa0ac22212938b with delete_type TABLET_DATA_DELETED (replaced by tablet b3ad595efe5241e8a4f40b16be2ccd63) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:49.220368 15686 leader_election.cc:290] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:49.223053 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c39cc816f4c438391bb4826ea1a508f" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:49.228214 14416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48575fa3f45349b9887bfc1cdfebc958. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.232009 15697 raft_consensus.cc:515] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.239043 15514 raft_consensus.cc:493] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:49.239542 15514 raft_consensus.cc:515] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.241966 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e05628011634149b17d298c54074d00" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:49.243104 14544 leader_election.cc:343] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:49.243868 15514 leader_election.cc:290] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:49.244580 14583 tablet_service.cc:1552] Processing DeleteTablet for tablet 080d1700df4a4df09acda65c6d021d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:49.249061 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e05628011634149b17d298c54074d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:49.252161 14274 tablet_service.cc:1552] Processing DeleteTablet for tablet e15483f539f24f1ba1b442663d66fd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:49.254251 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 9fac41e1f10446d884cac85c4044167b (DEFAULT_TABLE table=default.Table_4a99523c26ff417bbf24b0deb1b04b09 [id=61d1d967c6864029bac33248ffcbbced]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.255301 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fac41e1f10446d884cac85c4044167b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.256217 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_9294d4662790426dbb71410c2c874bc0" } new_table_name: "default.Table_6498d7f7c3504db994bd634f4d2b6ab6" modify_external_catalogs: true
I20251211 08:57:49.189654 14714 tablet_bootstrap.cc:492] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:49.259315 14273 tablet_service.cc:1505] Processing CreateTablet for tablet dfa418dbd71446c1988a1e543cf160d2 (DEFAULT_TABLE table=default.Table_5de14693e76648da88962b5448218b76 [id=41d7f3f78a404faaab21fdf2a88171e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.259866 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e05628011634149b17d298c54074d00" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:49.260161 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa418dbd71446c1988a1e543cf160d2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:49.262326 14542 leader_election.cc:343] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:49.262722 14542 leader_election.cc:304] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:49.264006 15670 raft_consensus.cc:2749] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:49.265882 14012 master_service.cc:954] ReplaceTablet: received request to replace tablet c03f976b926a4a6fba8b70e9df9a4160 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:49.268419 15030 tablet_bootstrap.cc:492] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:49.269021 15030 ts_tablet_manager.cc:1403] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.205s user 0.009s sys 0.004s
I20251211 08:57:49.269371 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7bce90f465c249f9a5ab2b250fb5610b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:49.270660 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bce90f465c249f9a5ab2b250fb5610b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:49.272871 14931 fs_manager.cc:714] Time spent opening directory manager: real 0.093s user 0.004s sys 0.003s
I20251211 08:57:49.278141 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f1453eb82ce2412c8208000d8ba7f240" } modify_external_catalogs: true
I20251211 08:57:49.278563 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f1453eb82ce2412c8208000d8ba7f240" } modify_external_catalogs: true
I20251211 08:57:49.281028 14586 tablet_service.cc:1505] Processing CreateTablet for tablet dfa418dbd71446c1988a1e543cf160d2 (DEFAULT_TABLE table=default.Table_5de14693e76648da88962b5448218b76 [id=41d7f3f78a404faaab21fdf2a88171e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.281944 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa418dbd71446c1988a1e543cf160d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.282263 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9905227c0a844d199362a442e7d630c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:49.283391 14544 leader_election.cc:343] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:49.283752 14544 leader_election.cc:304] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:49.284580 15514 raft_consensus.cc:2749] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:49.294613 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 7b6cdf1adae34ad2908233ce7323cf65 (DEFAULT_TABLE table=default.Table_9294d4662790426dbb71410c2c874bc0 [id=eec20447308f431092b2b3736ae03d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.295177 14422 tablet_service.cc:1552] Processing DeleteTablet for tablet e15483f539f24f1ba1b442663d66fd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:49.295578 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b6cdf1adae34ad2908233ce7323cf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.296322 14421 tablet_service.cc:1505] Processing CreateTablet for tablet dfa418dbd71446c1988a1e543cf160d2 (DEFAULT_TABLE table=default.Table_5de14693e76648da88962b5448218b76 [id=41d7f3f78a404faaab21fdf2a88171e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.297246 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa418dbd71446c1988a1e543cf160d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.303947 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 47e3c0a8e8344e3aa688d3d6a48c15ed (DEFAULT_TABLE table=default.Table_d9c25365b4374da1aab0e61d5148e89f [id=d50fd3c4b2a84b3588518a5548a6f4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.304903 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e3c0a8e8344e3aa688d3d6a48c15ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.307471 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 7b337f08cb0d4e78a611c03229252d54 (DEFAULT_TABLE table=default.Table_55467d689ec140f2a3ed64cc4ec0d806 [id=a1823ac371e54801bafcfdfc71d7901e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.308522 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b337f08cb0d4e78a611c03229252d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.309620 15697 leader_election.cc:290] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:49.295107 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 9fac41e1f10446d884cac85c4044167b (DEFAULT_TABLE table=default.Table_4a99523c26ff417bbf24b0deb1b04b09 [id=61d1d967c6864029bac33248ffcbbced]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.312886 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fac41e1f10446d884cac85c4044167b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.314381 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 47e3c0a8e8344e3aa688d3d6a48c15ed (DEFAULT_TABLE table=default.Table_d9c25365b4374da1aab0e61d5148e89f [id=d50fd3c4b2a84b3588518a5548a6f4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.318188 14278 tablet_service.cc:1505] Processing CreateTablet for tablet 72943bd4f76b48c9849e3e1668f38b59 (DEFAULT_TABLE table=default.Table_41bf1acc3ec84195b9b1034fcdbe1e42 [id=d433e3a81be5459488139283fe673760]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:49.323706 14385 leader_election.cc:343] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:49.331188 14385 leader_election.cc:343] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:49.340898 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_bd5960f04a874bee8826f8d460f1388b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:49.342429 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd5960f04a874bee8826f8d460f1388b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:49.343012 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49015c5d4f2b4d07b547a9df57c992eb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:49.353125 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_466402e26ef741c8bb19f291f23f54b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:49.355684 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_466402e26ef741c8bb19f291f23f54b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:49.372305 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 47e3c0a8e8344e3aa688d3d6a48c15ed (DEFAULT_TABLE table=default.Table_d9c25365b4374da1aab0e61d5148e89f [id=d50fd3c4b2a84b3588518a5548a6f4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.373807 15715 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:57:49.373586 15705 consensus_queue.cc:1048] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:49.377974 14931 fs_manager.cc:730] Time spent opening block manager: real 0.049s user 0.006s sys 0.003s
I20251211 08:57:49.374169 15697 consensus_queue.cc:1048] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:49.379714 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e3c0a8e8344e3aa688d3d6a48c15ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.385253 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e3c0a8e8344e3aa688d3d6a48c15ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.387099 15699 raft_consensus.cc:493] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:49.387481 15699 raft_consensus.cc:515] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.389180 15699 leader_election.cc:290] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:49.389719 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72943bd4f76b48c9849e3e1668f38b59. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:49.391709 14230 leader_election.cc:343] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 98d14ba15dcd47f8b9e8035b3c7d477c
I20251211 08:57:49.373809 14423 tablet_service.cc:1505] Processing CreateTablet for tablet 72943bd4f76b48c9849e3e1668f38b59 (DEFAULT_TABLE table=default.Table_41bf1acc3ec84195b9b1034fcdbe1e42 [id=d433e3a81be5459488139283fe673760]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.393163 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72943bd4f76b48c9849e3e1668f38b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.378748 14931 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal
uuid: "73702d54989f4959acfc4a6fdcede6ca"
format_stamp: "Formatted at 2025-12-11 08:57:08 on dist-test-slave-0dfs"
I20251211 08:57:49.395155 14931 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:57:49.396906 14722 tablet_bootstrap.cc:492] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:49.400776 14722 ts_tablet_manager.cc:1403] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.251s user 0.007s sys 0.004s
I20251211 08:57:49.404642 14722 raft_consensus.cc:359] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.407755 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 72943bd4f76b48c9849e3e1668f38b59 (DEFAULT_TABLE table=default.Table_41bf1acc3ec84195b9b1034fcdbe1e42 [id=d433e3a81be5459488139283fe673760]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:49.411652 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72943bd4f76b48c9849e3e1668f38b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:49.431273 14722 raft_consensus.cc:385] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:49.432209 14722 raft_consensus.cc:740] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:49.440640 14299 raft_consensus.cc:3060] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:49.447016 14722 consensus_queue.cc:260] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.418385 14904 ts_tablet_manager.cc:1929] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:49.450793 14904 log.cc:1199] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/f5b462cf21e3420c8e323a4ac4f04abe
I20251211 08:57:49.453474 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_064b00aac2ff440891fc0f5f0143a573" } new_table_name: "default.Table_e5b962b03e05467d95482d75eaff0a43" modify_external_catalogs: true
W20251211 08:57:49.458253 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.62:39015, user_credentials={real_user=slave}} blocked reactor thread for 48917.5us
I20251211 08:57:49.461163 14904 ts_tablet_manager.cc:1950] T f5b462cf21e3420c8e323a4ac4f04abe P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:49.462682 14722 ts_tablet_manager.cc:1434] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.060s user 0.005s sys 0.000s
I20251211 08:57:49.430619 14905 ts_tablet_manager.cc:1929] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:49.271518 15030 raft_consensus.cc:359] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.467497 15030 raft_consensus.cc:385] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:49.468123 15030 raft_consensus.cc:740] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:49.469046 15030 consensus_queue.cc:260] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.472149 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 391) took 2914 ms. Trace:
I20251211 08:57:49.472443 15299 rpcz_store.cc:276] 1211 08:57:46.557415 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.557519 (+ 104us) service_pool.cc:225] Handling call
1211 08:57:49.067395 (+2509876us) ts_tablet_manager.cc:1151] Deleting tablet faa96580439242948a66bbf00aeb836a
1211 08:57:49.067434 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.472100 (+404666us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2509359}
W20251211 08:57:49.480053 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b6cdf1adae34ad2908233ce7323cf65 consensus state has no leader
I20251211 08:57:49.481370 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet c03f976b926a4a6fba8b70e9df9a4160
I20251211 08:57:49.482553 14012 catalog_manager.cc:6724] ReplaceTablet: tablet c03f976b926a4a6fba8b70e9df9a4160 deleted and replaced by tablet 07099552b546444bab4d66238dd6d146
I20251211 08:57:49.483457 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 45c7d841bd3643a799a8b96e82d6d675
I20251211 08:57:49.487774 15030 ts_tablet_manager.cc:1434] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.218s user 0.005s sys 0.000s
I20251211 08:57:49.491075 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 45c7d841bd3643a799a8b96e82d6d675 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:49.491339 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 358) took 2785 ms. Trace:
I20251211 08:57:49.493134 14722 tablet_bootstrap.cc:492] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:49.494221 14905 log.cc:1199] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/dd14de369e084828afb8a924c46c649b
I20251211 08:57:49.494773 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet c03f976b926a4a6fba8b70e9df9a4160 with delete_type TABLET_DATA_DELETED (replaced by tablet 07099552b546444bab4d66238dd6d146) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:49.497599 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 45c7d841bd3643a799a8b96e82d6d675 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:49.499120 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet c03f976b926a4a6fba8b70e9df9a4160 with delete_type TABLET_DATA_DELETED (replaced by tablet 07099552b546444bab4d66238dd6d146) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:49.524324 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 49268.6us
I20251211 08:57:49.515853 14905 ts_tablet_manager.cc:1950] T dd14de369e084828afb8a924c46c649b P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:49.531694 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 306) took 6251 ms. Trace:
I20251211 08:57:49.491724 14904 rpcz_store.cc:276] 1211 08:57:46.705812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.706499 (+ 687us) service_pool.cc:225] Handling call
1211 08:57:49.052757 (+2346258us) ts_tablet_manager.cc:1151] Deleting tablet f5b462cf21e3420c8e323a4ac4f04abe
1211 08:57:49.052808 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.189135 (+136327us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54001a0c54. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f141cc5a1 00007f0f1426afe2 00007f0f1468d2ec 00007f0f152e431a 00007f0f15314f80 00007f0f152fb54d 00007f0f152fb4f9 00007f0f152fb489 00007f0f152fb451 00007f0f152fa67d 00000000005aa2b4
1211 08:57:49.202723 (+ 13588us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.418189 (+215466us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.491268 (+ 73079us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":292114688,"tablet-delete.queue_time_us":2345582}
I20251211 08:57:49.539746 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet f5b462cf21e3420c8e323a4ac4f04abe (table default.Table_e444843f69064fbb9c2a6d0671fd9a2a [id=6c43f429406443d7b5834905929dcbd4]) successfully deleted
I20251211 08:57:49.540896 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 359) took 2809 ms. Trace:
I20251211 08:57:49.541596 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_cd9d0dad407f478f869e3d7654830da1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:49.543027 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd9d0dad407f478f869e3d7654830da1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:49.546988 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_70f3ebdd01dc4e75bc85ab29528865fc" } modify_external_catalogs: true
I20251211 08:57:49.547461 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_70f3ebdd01dc4e75bc85ab29528865fc" } modify_external_catalogs: true
I20251211 08:57:49.549474 14714 tablet_bootstrap.cc:654] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:49.383082 14385 leader_election.cc:343] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:49.550262 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 167297us
I20251211 08:57:49.552525 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet c03f976b926a4a6fba8b70e9df9a4160 with delete_type TABLET_DATA_DELETED (replaced by tablet 07099552b546444bab4d66238dd6d146) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:49.547796 14904 rpcz_store.cc:276] 1211 08:57:46.732465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.737088 (+ 4623us) service_pool.cc:225] Handling call
1211 08:57:49.540561 (+2803473us) ts_tablet_manager.cc:1151] Deleting tablet 9821822304c948f3bfcf5d52901371a4
1211 08:57:49.540598 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.540849 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2801954}
I20251211 08:57:49.581599 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 367) took 2476 ms. Trace:
I20251211 08:57:49.581884 14904 rpcz_store.cc:276] 1211 08:57:47.104881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.105247 (+ 366us) service_pool.cc:225] Handling call
1211 08:57:49.581274 (+2476027us) ts_tablet_manager.cc:1151] Deleting tablet 145096302ca24d7e96f2fec45960b18a
1211 08:57:49.581325 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.581550 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2474270}
I20251211 08:57:49.533208 15030 tablet_bootstrap.cc:492] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:49.587230 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 393) took 2856 ms. Trace:
I20251211 08:57:49.587564 15299 rpcz_store.cc:276] 1211 08:57:46.731317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.772560 (+ 41243us) service_pool.cc:225] Handling call
1211 08:57:49.473146 (+2700586us) ts_tablet_manager.cc:1151] Deleting tablet 9821822304c948f3bfcf5d52901371a4
1211 08:57:49.473183 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.587179 (+113996us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2680246}
I20251211 08:57:49.588500 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 372) took 2016 ms. Trace:
I20251211 08:57:49.588794 14904 rpcz_store.cc:276] 1211 08:57:47.572449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.666838 (+ 94389us) service_pool.cc:225] Handling call
1211 08:57:49.582635 (+1915797us) ts_tablet_manager.cc:1151] Deleting tablet 51c3f5eaef784e868e565f81bd51a5e4
1211 08:57:49.582666 (+ 31us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.588452 (+ 5786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1914894}
I20251211 08:57:49.583247 14905 rpcz_store.cc:276] 1211 08:57:43.280746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.280860 (+ 114us) service_pool.cc:225] Handling call
1211 08:57:49.132160 (+5851300us) ts_tablet_manager.cc:1151] Deleting tablet dd14de369e084828afb8a924c46c649b
1211 08:57:49.132194 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.387832 (+255638us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.430436 (+ 42604us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.531642 (+101206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5841943}
I20251211 08:57:49.589838 14905 tablet_replica.cc:333] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:49.589941 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 374) took 1880 ms. Trace:
I20251211 08:57:49.590204 14904 rpcz_store.cc:276] 1211 08:57:47.709744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.709842 (+ 98us) service_pool.cc:225] Handling call
1211 08:57:49.589517 (+1879675us) ts_tablet_manager.cc:1151] Deleting tablet 8c1e2d6ba1014d00a6df8c6353cff3ab
1211 08:57:49.589548 (+ 31us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.589897 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879205}
I20251211 08:57:49.590557 14905 raft_consensus.cc:2243] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:49.590972 14905 raft_consensus.cc:2272] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:49.591368 14904 tablet_replica.cc:333] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:49.592075 14904 raft_consensus.cc:2243] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:49.592813 14905 ts_tablet_manager.cc:1916] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:49.611843 15030 tablet_bootstrap.cc:654] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:49.619053 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 45c7d841bd3643a799a8b96e82d6d675 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:49.616356 14905 ts_tablet_manager.cc:1929] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:49.626995 14905 log.cc:1199] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/f909d82e0136470a810e1063d6769868
I20251211 08:57:49.627696 14905 ts_tablet_manager.cc:1950] T f909d82e0136470a810e1063d6769868 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:49.629437 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 313) took 5927 ms. Trace:
I20251211 08:57:49.629765 14905 rpcz_store.cc:276] 1211 08:57:43.701902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.703128 (+ 1226us) service_pool.cc:225] Handling call
1211 08:57:49.589707 (+5886579us) ts_tablet_manager.cc:1151] Deleting tablet f909d82e0136470a810e1063d6769868
1211 08:57:49.589745 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.608110 (+ 18365us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.616181 (+ 8071us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.629376 (+ 13195us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5886072}
I20251211 08:57:49.630842 14905 tablet_replica.cc:333] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:49.631673 14905 raft_consensus.cc:2243] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:49.632158 14905 raft_consensus.cc:2272] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:49.639758 14905 ts_tablet_manager.cc:1916] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:49.572183 15648 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.017s sys 0.033s
I20251211 08:57:49.646482 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 401) took 2563 ms. Trace:
I20251211 08:57:49.646822 15299 rpcz_store.cc:276] 1211 08:57:47.082995 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.084131 (+ 1136us) service_pool.cc:225] Handling call
1211 08:57:49.588863 (+2504732us) ts_tablet_manager.cc:1151] Deleting tablet 145096302ca24d7e96f2fec45960b18a
1211 08:57:49.588915 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.646431 (+ 57516us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2245458}
I20251211 08:57:49.649132 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98d14ba15dcd47f8b9e8035b3c7d477c" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:49.640944 15648 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.744s user 0.017s sys 0.033s
I20251211 08:57:49.652098 14904 raft_consensus.cc:2272] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:49.652190 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 407) took 2063 ms. Trace:
I20251211 08:57:49.652606 15299 rpcz_store.cc:276] 1211 08:57:47.589203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.601070 (+ 11867us) service_pool.cc:225] Handling call
1211 08:57:49.647700 (+2046630us) ts_tablet_manager.cc:1151] Deleting tablet 51c3f5eaef784e868e565f81bd51a5e4
1211 08:57:49.648662 (+ 962us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.652132 (+ 3470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1975723}
I20251211 08:57:49.653798 15299 tablet_replica.cc:333] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:49.654486 14904 ts_tablet_manager.cc:1916] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:49.659976 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c39cc816f4c438391bb4826ea1a508f" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:49.664204 15030 tablet_bootstrap.cc:492] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:49.665051 15030 ts_tablet_manager.cc:1403] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.147s user 0.008s sys 0.000s
I20251211 08:57:49.666894 14722 tablet_bootstrap.cc:654] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:49.671267 13997 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 7b6cdf1adae34ad2908233ce7323cf65 (table default.Table_6498d7f7c3504db994bd634f4d2b6ab6 [id=eec20447308f431092b2b3736ae03d47], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b6cdf1adae34ad2908233ce7323cf65 consensus state has no leader
W20251211 08:57:49.671567 13902 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 44667.6us
I20251211 08:57:49.675431 15030 raft_consensus.cc:359] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.676064 15030 raft_consensus.cc:385] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:49.676304 15030 raft_consensus.cc:740] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:49.676551 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet c341b762afd8440bb7198ca134c8dffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:37 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:49.676644 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet faa96580439242948a66bbf00aeb836a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faa96580439242948a66bbf00aeb836a already in progress: creating tablet
I20251211 08:57:49.681334 15030 consensus_queue.cc:260] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.682447 14010 master_service.cc:954] ReplaceTablet: received request to replace tablet 4ab48f6288494005ad42208e2e97ddf2 from {username='slave'} at 127.0.0.1:55614
W20251211 08:57:49.682554 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 9821822304c948f3bfcf5d52901371a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9821822304c948f3bfcf5d52901371a4 already in progress: creating tablet
W20251211 08:57:49.696390 14382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.001s sys 0.027s
W20251211 08:57:49.696669 14382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.797s user 0.001s sys 0.027s
W20251211 08:57:49.697567 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 145096302ca24d7e96f2fec45960b18a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 145096302ca24d7e96f2fec45960b18a already in progress: creating tablet
W20251211 08:57:49.699981 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 51c3f5eaef784e868e565f81bd51a5e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51c3f5eaef784e868e565f81bd51a5e4 already in progress: creating tablet
I20251211 08:57:49.702340 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49015c5d4f2b4d07b547a9df57c992eb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:49.702916 14606 raft_consensus.cc:3060] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:49.703846 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45c7d841bd3643a799a8b96e82d6d675" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:49.704306 14604 raft_consensus.cc:3060] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:49.689656 15514 raft_consensus.cc:493] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:49.705497 15514 raft_consensus.cc:515] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.707142 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da57b72207414512b9744b283bc869fe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:49.707580 14600 raft_consensus.cc:3060] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:49.715282 15514 leader_election.cc:290] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:49.716275 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb2ff90b54544420b1aad9d2ab4132ff" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:49.717547 14542 leader_election.cc:343] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:49.726593 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb2ff90b54544420b1aad9d2ab4132ff" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:49.737368 15697 consensus_queue.cc:1048] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:49.737797 15710 consensus_queue.cc:1048] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:49.702353 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca5eebf2ce3848c79ee2fc21ad2b1486 consensus state has no leader
W20251211 08:57:49.740353 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c1e2d6ba1014d00a6df8c6353cff3ab already in progress: creating tablet
W20251211 08:57:49.695135 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 9821822304c948f3bfcf5d52901371a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9821822304c948f3bfcf5d52901371a4 already in progress: creating tablet
W20251211 08:57:49.746706 13998 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 51643.3us
I20251211 08:57:49.752270 14604 raft_consensus.cc:2468] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:49.711182 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet dd14de369e084828afb8a924c46c649b (table default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]) successfully deleted
W20251211 08:57:49.756995 13997 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 45870.3us
I20251211 08:57:49.758270 14600 raft_consensus.cc:2468] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:49.760056 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet f909d82e0136470a810e1063d6769868 (table default.Table_c15367b621c144348ec8511d2947fec6 [id=ea01e214ed314a26a0de20d34e37863b]) successfully deleted
I20251211 08:57:49.758620 14382 leader_election.cc:304] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:49.760977 15710 raft_consensus.cc:2804] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:49.761482 15710 raft_consensus.cc:697] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
W20251211 08:57:49.758157 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 145096302ca24d7e96f2fec45960b18a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 145096302ca24d7e96f2fec45960b18a already in progress: creating tablet
I20251211 08:57:49.762270 15710 consensus_queue.cc:237] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:49.763510 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 51c3f5eaef784e868e565f81bd51a5e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51c3f5eaef784e868e565f81bd51a5e4 already in progress: creating tablet
I20251211 08:57:49.768709 14606 raft_consensus.cc:2468] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:49.778661 15696 raft_consensus.cc:493] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:49.779246 15696 raft_consensus.cc:515] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.782106 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ab48f6288494005ad42208e2e97ddf2" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:49.783468 14385 leader_election.cc:343] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:49.785068 15696 leader_election.cc:290] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:49.785893 14722 tablet_bootstrap.cc:492] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:49.786314 14722 ts_tablet_manager.cc:1403] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.293s user 0.009s sys 0.000s
I20251211 08:57:49.788539 14714 tablet_bootstrap.cc:492] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:49.788431 14722 raft_consensus.cc:359] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.789000 14722 raft_consensus.cc:385] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:49.789223 14722 raft_consensus.cc:740] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:49.789333 14011 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_bf148c361a2346e88e4c9e4430b1abca" } new_table_name: "default.Table_34b8d290de5444d49bed8a5b6ec99f05" modify_external_catalogs: true
I20251211 08:57:49.789755 14722 consensus_queue.cc:260] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.792526 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a813bbe569514a5ea2be6f06f90629ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:49.796226 14714 ts_tablet_manager.cc:1403] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.607s user 0.014s sys 0.012s
I20251211 08:57:49.799053 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b7ff902a532d4e7697811e15bcafde66" } modify_external_catalogs: true
I20251211 08:57:49.799045 14714 raft_consensus.cc:359] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.799721 14714 raft_consensus.cc:385] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:49.799674 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b7ff902a532d4e7697811e15bcafde66" } modify_external_catalogs: true
I20251211 08:57:49.799954 14714 raft_consensus.cc:740] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:49.806133 15299 raft_consensus.cc:2243] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
W20251211 08:57:49.794677 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a813bbe569514a5ea2be6f06f90629ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:49.821587 14714 consensus_queue.cc:260] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.836941 15299 raft_consensus.cc:2272] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:49.838572 14714 ts_tablet_manager.cc:1434] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20251211 08:57:49.838814 15299 ts_tablet_manager.cc:1916] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:49.839581 14714 tablet_bootstrap.cc:492] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:49.843397 14231 leader_election.cc:343] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 3c2fb2d9564c43ee84b8dfcea8ec54b0
I20251211 08:57:49.843807 14231 leader_election.cc:304] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:49.844455 15699 raft_consensus.cc:3060] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:49.851078 15699 raft_consensus.cc:2749] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:49.856186 14231 leader_election.cc:343] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:49.856590 14231 leader_election.cc:304] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:49.857264 14722 ts_tablet_manager.cc:1434] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.071s user 0.003s sys 0.003s
I20251211 08:57:49.858110 14722 tablet_bootstrap.cc:492] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:49.858004 14231 leader_election.cc:304] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:49.862152 15699 raft_consensus.cc:2749] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:49.862807 15699 raft_consensus.cc:2749] T 8c39cc816f4c438391bb4826ea1a508f P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:49.868969 14714 tablet_bootstrap.cc:654] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:49.900345 14714 tablet_bootstrap.cc:492] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
W20251211 08:57:49.900348 14544 leader_election.cc:343] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:49.900735 14714 ts_tablet_manager.cc:1403] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.003s
I20251211 08:57:49.900852 14544 leader_election.cc:304] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:49.903359 15740 raft_consensus.cc:2749] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:49.903247 14714 raft_consensus.cc:359] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.903843 14714 raft_consensus.cc:385] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:49.904090 14714 raft_consensus.cc:740] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:49.908047 15299 ts_tablet_manager.cc:1929] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:49.913635 14722 tablet_bootstrap.cc:654] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:49.922068 15740 raft_consensus.cc:493] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:49.922564 15740 raft_consensus.cc:515] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.924721 15740 leader_election.cc:290] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:49.926483 15030 ts_tablet_manager.cc:1434] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.261s user 0.007s sys 0.000s
I20251211 08:57:49.951647 15030 tablet_bootstrap.cc:492] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:49.952381 15299 log.cc:1199] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/f938359354b545cfbac560b7d7888b9f
I20251211 08:57:49.953199 15299 ts_tablet_manager.cc:1950] T f938359354b545cfbac560b7d7888b9f P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:49.915158 14714 consensus_queue.cc:260] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.955382 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 408) took 2278 ms. Trace:
I20251211 08:57:49.956845 15299 rpcz_store.cc:276] 1211 08:57:47.676699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.678547 (+ 1848us) service_pool.cc:225] Handling call
1211 08:57:49.653611 (+1975064us) ts_tablet_manager.cc:1151] Deleting tablet f938359354b545cfbac560b7d7888b9f
1211 08:57:49.653660 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.807859 (+154199us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b5400113d54. stack: 00007efeecdf4083 00007efeecdf3f80 00007efeec969574 0000000000508744 00000000005086d9 00007efef3e3b5a1 00007efef3ed9fe2 00007efef42fc2ec 00007efef4f5331a 00007efef4f83f80 00007efef4f6a54d 00007efef4f6a4f9 00007efef4f6a489 00007efef4f6a451 00007efef4f6967d 00000000005aa2b4
1211 08:57:49.845638 (+ 37779us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.907791 (+ 62153us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.955331 (+ 47540us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38993,"spinlock_wait_cycles":258855296,"tablet-delete.queue_time_us":1974510}
I20251211 08:57:49.916657 14904 ts_tablet_manager.cc:1929] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:49.942838 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c2fb2d9564c43ee84b8dfcea8ec54b0" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:49.965183 14299 raft_consensus.cc:2468] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:49.966913 14904 log.cc:1199] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/f938359354b545cfbac560b7d7888b9f
I20251211 08:57:49.967684 14904 ts_tablet_manager.cc:1950] T f938359354b545cfbac560b7d7888b9f P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:49.969211 14544 leader_election.cc:304] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:49.970468 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet e62ae5b540aa4680b4d18fee29c39e64
I20251211 08:57:49.783679 14905 ts_tablet_manager.cc:1929] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:49.973946 14905 log.cc:1199] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/5f7775994f684326b6312c802f6b084d
I20251211 08:57:49.974824 14905 ts_tablet_manager.cc:1950] T 5f7775994f684326b6312c802f6b084d P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:49.976123 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet e62ae5b540aa4680b4d18fee29c39e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:49.977613 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_bbc9ffbf679e4968b71046d57d9aae32" } new_table_name: "default.Table_60479fec7ce247e687a133fd0dbad564" modify_external_catalogs: true
I20251211 08:57:49.979808 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_c57ef6202bd14b8c9ff0e4a6bbe161b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:49.981261 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c57ef6202bd14b8c9ff0e4a6bbe161b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:49.986501 14722 tablet_bootstrap.cc:492] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:49.989732 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_5e9e1c179cbb4953b47e0c0400ee35fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:49.990180 14722 ts_tablet_manager.cc:1403] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.132s user 0.008s sys 0.003s
W20251211 08:57:49.991297 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e9e1c179cbb4953b47e0c0400ee35fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:49.992270 14722 raft_consensus.cc:359] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:49.995312 15752 tablet.cc:1722] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:49.999564 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_1df60c18431f4fcbb050908676608f64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:50.000233 14722 raft_consensus.cc:385] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:50.000504 14722 raft_consensus.cc:740] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
W20251211 08:57:50.001348 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1df60c18431f4fcbb050908676608f64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:50.001029 14722 consensus_queue.cc:260] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.017102 14722 ts_tablet_manager.cc:1434] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.027s user 0.000s sys 0.005s
I20251211 08:57:49.920940 14382 leader_election.cc:304] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
W20251211 08:57:50.032927 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 111953us
I20251211 08:57:50.034565 14382 leader_election.cc:304] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:50.040432 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet f938359354b545cfbac560b7d7888b9f (table default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]) successfully deleted
I20251211 08:57:50.042325 14421 tablet_service.cc:1552] Processing DeleteTablet for tablet e62ae5b540aa4680b4d18fee29c39e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:50.048171 15743 raft_consensus.cc:493] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:50.048599 15743 raft_consensus.cc:515] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.050462 15743 leader_election.cc:290] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.055866 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b9ae463731d4828afc02039e0d161b5" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:49.961616 14714 ts_tablet_manager.cc:1434] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.061s user 0.006s sys 0.001s
W20251211 08:57:50.061195 14231 leader_election.cc:343] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.061640 14714 tablet_bootstrap.cc:492] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:50.064581 15648 raft_consensus.cc:2804] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:50.022336 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet f785a3cc2e8145eebae7e9c2885b0b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:50.020277 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ab48f6288494005ad42208e2e97ddf2" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:50.049901 15738 raft_consensus.cc:493] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:50.051906 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b9ae463731d4828afc02039e0d161b5" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:50.072219 14608 raft_consensus.cc:2468] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:50.044296 15514 raft_consensus.cc:493] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:50.073124 14230 leader_election.cc:304] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:50.074203 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 314) took 6343 ms. Trace:
I20251211 08:57:50.074542 14905 rpcz_store.cc:276] 1211 08:57:43.730730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.733280 (+ 2550us) service_pool.cc:225] Handling call
1211 08:57:49.630691 (+5897411us) ts_tablet_manager.cc:1151] Deleting tablet 5f7775994f684326b6312c802f6b084d
1211 08:57:49.630725 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.654242 (+ 23517us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.783442 (+129200us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.074143 (+290701us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":232,"tablet-delete.queue_time_us":5896815}
I20251211 08:57:50.079159 15743 raft_consensus.cc:2804] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:50.079457 15743 raft_consensus.cc:493] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:50.079733 15743 raft_consensus.cc:3060] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.091418 14722 tablet_bootstrap.cc:492] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:50.090754 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 373) took 2414 ms. Trace:
I20251211 08:57:50.095095 15751 tablet.cc:1722] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:57:50.095297 14904 rpcz_store.cc:276] 1211 08:57:47.676279 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.727093 (+ 50814us) service_pool.cc:225] Handling call
1211 08:57:49.590884 (+1863791us) ts_tablet_manager.cc:1151] Deleting tablet f938359354b545cfbac560b7d7888b9f
1211 08:57:49.591269 (+ 385us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:49.690009 (+ 98740us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:49.916362 (+226353us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.090703 (+174341us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1863254}
I20251211 08:57:50.096412 14904 tablet_replica.cc:333] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:50.097004 14904 raft_consensus.cc:2243] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:50.097450 14904 raft_consensus.cc:2272] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:50.099785 14904 ts_tablet_manager.cc:1916] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:50.104274 14722 tablet_bootstrap.cc:654] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:50.107285 15650 raft_consensus.cc:2804] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:50.107789 15650 raft_consensus.cc:697] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:50.108855 15650 consensus_queue.cc:237] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.114933 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 409) took 2406 ms. Trace:
I20251211 08:57:50.115304 15299 rpcz_store.cc:276] 1211 08:57:47.708177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.708491 (+ 314us) service_pool.cc:225] Handling call
1211 08:57:50.101905 (+2393414us) ts_tablet_manager.cc:1151] Deleting tablet 8c1e2d6ba1014d00a6df8c6353cff3ab
1211 08:57:50.114347 (+ 12442us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.114730 (+ 383us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2388241}
W20251211 08:57:50.117027 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c1e2d6ba1014d00a6df8c6353cff3ab already in progress: creating tablet
I20251211 08:57:50.126941 15299 tablet_replica.cc:333] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:50.128191 15299 raft_consensus.cc:2243] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:50.128748 15299 raft_consensus.cc:2272] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:50.129627 15743 raft_consensus.cc:515] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.130514 15299 ts_tablet_manager.cc:1916] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:50.133976 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b9ae463731d4828afc02039e0d161b5" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:50.134236 15696 raft_consensus.cc:493] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:50.134752 15696 raft_consensus.cc:515] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:50.135317 14231 leader_election.cc:343] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.136543 15696 leader_election.cc:290] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.136682 15743 leader_election.cc:290] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.137811 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6d0fef50524476dbd47fc70e4bda319" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:50.138963 14385 leader_election.cc:343] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.098752 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet f938359354b545cfbac560b7d7888b9f (table default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]) successfully deleted
I20251211 08:57:50.170864 15648 raft_consensus.cc:697] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:50.171304 14012 master_service.cc:954] ReplaceTablet: received request to replace tablet ccce5a446ac9460c9c9b7de72d875d95 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:50.172070 15648 consensus_queue.cc:237] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.181165 14722 tablet_bootstrap.cc:492] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:50.181643 14722 ts_tablet_manager.cc:1403] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.002s
I20251211 08:57:50.184195 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 5f7775994f684326b6312c802f6b084d (table default.Table_0ff1ebe3d6b144cc94137467d38b30bb [id=e5e62d49ddb749a29892b61cd4615da7]) successfully deleted
I20251211 08:57:50.187870 14722 raft_consensus.cc:359] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.188629 14722 raft_consensus.cc:385] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:50.188894 14722 raft_consensus.cc:740] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:50.189157 14714 tablet_bootstrap.cc:654] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:50.189597 14722 consensus_queue.cc:260] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.193758 15699 raft_consensus.cc:493] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:50.194137 15699 raft_consensus.cc:515] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.195964 15699 leader_election.cc:290] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.197576 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da52cace2aab4f9dacef18ec5fa56aaf" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:50.198822 14231 leader_election.cc:343] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.022120 15739 raft_consensus.cc:2804] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:57:50.065079 15030 tablet_bootstrap.cc:654] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:50.185501 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet e62ae5b540aa4680b4d18fee29c39e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:50.225581 15739 raft_consensus.cc:493] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:50.226272 15739 raft_consensus.cc:3060] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:57:50.229005 14714 tablet_bootstrap.cc:492] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:50.229368 14714 ts_tablet_manager.cc:1403] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.168s user 0.008s sys 0.008s
I20251211 08:57:50.231061 15743 raft_consensus.cc:493] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:50.231462 14714 raft_consensus.cc:359] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.232245 14714 raft_consensus.cc:385] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:50.232535 14714 raft_consensus.cc:740] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:50.231964 15743 raft_consensus.cc:515] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.235749 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cbd18e927c843e7b0aee7e0ccb1b06d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:50.195603 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b9ae463731d4828afc02039e0d161b5" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
W20251211 08:57:50.196489 14542 leader_election.cc:343] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 3c2fb2d9564c43ee84b8dfcea8ec54b0
I20251211 08:57:50.236811 14608 raft_consensus.cc:3060] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:50.237040 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 89589.1us
I20251211 08:57:50.210733 14602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6d0fef50524476dbd47fc70e4bda319" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:50.237999 14714 consensus_queue.cc:260] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.241472 15743 leader_election.cc:290] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.242873 14904 ts_tablet_manager.cc:1929] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251211 08:57:50.243213 14904 log.cc:1199] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/1a2dcb9ac62e4aac885d638f4e2b072e
I20251211 08:57:50.243901 14904 ts_tablet_manager.cc:1950] T 1a2dcb9ac62e4aac885d638f4e2b072e P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:50.243898 14444 raft_consensus.cc:2468] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:50.244655 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 375) took 2291 ms. Trace:
I20251211 08:57:50.244940 14904 rpcz_store.cc:276] 1211 08:57:47.953724 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.954838 (+ 1114us) service_pool.cc:225] Handling call
1211 08:57:50.096272 (+2141434us) ts_tablet_manager.cc:1151] Deleting tablet 1a2dcb9ac62e4aac885d638f4e2b072e
1211 08:57:50.096310 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.154670 (+ 58360us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.242567 (+ 87897us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.244602 (+ 2035us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2053743}
I20251211 08:57:50.246068 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 1a2dcb9ac62e4aac885d638f4e2b072e (table default.Table_0c2ec37766b24f539c7ff1e0aa4ac9c0 [id=1673af33c2f846c7bf092303eff6e8ad]) successfully deleted
I20251211 08:57:50.246372 14722 ts_tablet_manager.cc:1434] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.064s user 0.007s sys 0.000s
I20251211 08:57:50.247601 14231 leader_election.cc:304] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:50.248471 15743 raft_consensus.cc:2804] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:50.248864 15743 raft_consensus.cc:493] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:50.249420 15743 raft_consensus.cc:3060] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.249565 14714 ts_tablet_manager.cc:1434] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20251211 08:57:50.250494 14714 tablet_bootstrap.cc:492] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:50.250324 14722 tablet_bootstrap.cc:492] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:50.185103 14600 raft_consensus.cc:2468] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:50.183044 15738 raft_consensus.cc:515] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.269721 15738 leader_election.cc:290] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:50.180894 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 315) took 6391 ms. Trace:
I20251211 08:57:50.266021 14602 raft_consensus.cc:2468] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:50.271047 14905 rpcz_store.cc:276] 1211 08:57:43.789728 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.789839 (+ 111us) service_pool.cc:225] Handling call
1211 08:57:50.075553 (+6285714us) ts_tablet_manager.cc:1151] Deleting tablet 74c8dd2b036c42a0bec79d3637f00ef4
1211 08:57:50.180553 (+105000us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.180818 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6285222}
I20251211 08:57:50.272639 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 316) took 6452 ms. Trace:
I20251211 08:57:50.273020 14905 rpcz_store.cc:276] 1211 08:57:43.819865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:43.820553 (+ 688us) service_pool.cc:225] Handling call
1211 08:57:50.272149 (+6451596us) ts_tablet_manager.cc:1151] Deleting tablet c112dfcc301648cda6f0f038b1f2af83
1211 08:57:50.272191 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.272585 (+ 394us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6451090}
I20251211 08:57:50.274493 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 319) took 6044 ms. Trace:
I20251211 08:57:50.275040 15299 ts_tablet_manager.cc:1929] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251211 08:57:50.275485 15299 log.cc:1199] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/1a2dcb9ac62e4aac885d638f4e2b072e
W20251211 08:57:50.275825 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet c112dfcc301648cda6f0f038b1f2af83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c112dfcc301648cda6f0f038b1f2af83 already in progress: creating tablet
I20251211 08:57:50.279459 15299 ts_tablet_manager.cc:1950] T 1a2dcb9ac62e4aac885d638f4e2b072e P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:50.281468 14931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:57:50.284847 14905 rpcz_store.cc:276] 1211 08:57:44.229739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.230021 (+ 282us) service_pool.cc:225] Handling call
1211 08:57:50.274091 (+6044070us) ts_tablet_manager.cc:1151] Deleting tablet 4c95565ce3954cc598d7cbd45321d7c9
1211 08:57:50.274142 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.274435 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6043566}
I20251211 08:57:50.281651 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 410) took 2328 ms. Trace:
W20251211 08:57:50.286720 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 4c95565ce3954cc598d7cbd45321d7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c95565ce3954cc598d7cbd45321d7c9 already in progress: creating tablet
I20251211 08:57:50.287140 15299 rpcz_store.cc:276] 1211 08:57:47.953313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.955475 (+ 2162us) service_pool.cc:225] Handling call
1211 08:57:50.116632 (+2161157us) ts_tablet_manager.cc:1151] Deleting tablet 1a2dcb9ac62e4aac885d638f4e2b072e
1211 08:57:50.126783 (+ 10151us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.215576 (+ 88793us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.273695 (+ 58119us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.281599 (+ 7904us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2130257}
I20251211 08:57:50.289700 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 320) took 6055 ms. Trace:
I20251211 08:57:50.289960 14905 rpcz_store.cc:276] 1211 08:57:44.233908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.234028 (+ 120us) service_pool.cc:225] Handling call
1211 08:57:50.289429 (+6055401us) ts_tablet_manager.cc:1151] Deleting tablet ed0e90efe651499c929b1fb992712b96
1211 08:57:50.289464 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.289659 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6054749}
I20251211 08:57:50.290453 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 1a2dcb9ac62e4aac885d638f4e2b072e (table default.Table_0c2ec37766b24f539c7ff1e0aa4ac9c0 [id=1673af33c2f846c7bf092303eff6e8ad]) successfully deleted
W20251211 08:57:50.291026 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet ed0e90efe651499c929b1fb992712b96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed0e90efe651499c929b1fb992712b96 already in progress: creating tablet
I20251211 08:57:50.291951 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 321) took 6004 ms. Trace:
I20251211 08:57:50.292222 14905 rpcz_store.cc:276] 1211 08:57:44.287423 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.287541 (+ 118us) service_pool.cc:225] Handling call
1211 08:57:50.291661 (+6004120us) ts_tablet_manager.cc:1151] Deleting tablet 07f983604bfa474f9c9ce0c982d738fc
1211 08:57:50.291699 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.291907 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6003612}
W20251211 08:57:50.292934 14542 consensus_peers.cc:597] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a2dcb9ac62e4aac885d638f4e2b072e. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:50.293030 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341b762afd8440bb7198ca134c8dffe" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:50.293589 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 07f983604bfa474f9c9ce0c982d738fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07f983604bfa474f9c9ce0c982d738fc already in progress: creating tablet
I20251211 08:57:50.298848 14382 leader_election.cc:304] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:50.299062 14714 tablet_bootstrap.cc:654] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:50.299973 14722 tablet_bootstrap.cc:654] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:50.300274 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341b762afd8440bb7198ca134c8dffe" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:50.301194 15648 raft_consensus.cc:2804] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:50.301513 15648 raft_consensus.cc:493] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:50.301805 15648 raft_consensus.cc:3060] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.303285 14382 leader_election.cc:304] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:50.311142 15743 raft_consensus.cc:515] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.312745 14931 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:57:50.314947 15650 raft_consensus.cc:2804] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:50.315286 15650 raft_consensus.cc:493] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:50.315582 15650 raft_consensus.cc:3060] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.320777 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a522cb7af78a49aaaa070d2ee83bdb19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:50.322422 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a522cb7af78a49aaaa070d2ee83bdb19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:50.322911 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 415) took 1811 ms. Trace:
I20251211 08:57:50.322825 15650 raft_consensus.cc:515] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.323276 15299 rpcz_store.cc:276] 1211 08:57:48.511152 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.515550 (+ 4398us) service_pool.cc:225] Handling call
1211 08:57:50.299088 (+1783538us) ts_tablet_manager.cc:1151] Deleting tablet f3cf2b4d14ba45d79e52b6fdb0497d58
1211 08:57:50.299158 (+ 70us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.322821 (+ 23663us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1775863}
I20251211 08:57:50.324361 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a813bbe569514a5ea2be6f06f90629ba" } new_table_name: "default.Table_479b5ad717dc450689b1fb6ae89169bf" modify_external_catalogs: true
W20251211 08:57:50.324522 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f3cf2b4d14ba45d79e52b6fdb0497d58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3cf2b4d14ba45d79e52b6fdb0497d58 already in progress: creating tablet
I20251211 08:57:50.325486 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6d0fef50524476dbd47fc70e4bda319" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:50.326671 14385 leader_election.cc:343] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.327164 15648 raft_consensus.cc:515] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.327775 15650 leader_election.cc:290] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.329042 15648 leader_election.cc:290] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.329872 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ab48f6288494005ad42208e2e97ddf2" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:50.330920 14385 leader_election.cc:343] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.331635 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 416) took 1820 ms. Trace:
I20251211 08:57:50.333065 15299 rpcz_store.cc:276] 1211 08:57:48.511452 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.516226 (+ 4774us) service_pool.cc:225] Handling call
1211 08:57:50.331173 (+1814947us) ts_tablet_manager.cc:1151] Deleting tablet aadc3c1ecd974ae3a0d491bd209047e6
1211 08:57:50.331254 (+ 81us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.331577 (+ 323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1814486}
I20251211 08:57:50.331743 14931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:57:50.339176 15743 leader_election.cc:290] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.333412 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cbd18e927c843e7b0aee7e0ccb1b06d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:50.340158 14445 raft_consensus.cc:3060] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.347978 14445 raft_consensus.cc:2468] T 4cbd18e927c843e7b0aee7e0ccb1b06d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:50.348966 14231 leader_election.cc:304] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:50.349402 15299 tablet_replica.cc:333] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:50.350172 15299 raft_consensus.cc:2243] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:50.350646 15299 raft_consensus.cc:2272] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:50.355885 14714 tablet_bootstrap.cc:492] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:50.357280 14714 ts_tablet_manager.cc:1403] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.107s user 0.005s sys 0.006s
I20251211 08:57:50.359509 14714 raft_consensus.cc:359] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.294551 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 326) took 5852 ms. Trace:
W20251211 08:57:50.295181 14542 leader_election.cc:343] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:50.363185 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 68072.6us
I20251211 08:57:50.308573 14602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da52cace2aab4f9dacef18ec5fa56aaf" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:50.365614 14714 raft_consensus.cc:385] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:50.360941 15299 ts_tablet_manager.cc:1916] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:50.366019 15739 raft_consensus.cc:515] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:50.309886 14544 consensus_peers.cc:597] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a2dcb9ac62e4aac885d638f4e2b072e. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:50.345482 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cbd18e927c843e7b0aee7e0ccb1b06d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:50.369963 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 60183.8us
I20251211 08:57:50.370169 14600 raft_consensus.cc:2468] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
W20251211 08:57:50.370466 14544 leader_election.cc:343] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.370833 14544 leader_election.cc:304] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:50.371512 15670 raft_consensus.cc:2749] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:50.373003 14714 raft_consensus.cc:740] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
W20251211 08:57:50.373015 14230 leader_election.cc:343] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.373390 14230 leader_election.cc:304] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:50.373513 14602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6d0fef50524476dbd47fc70e4bda319" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:50.374006 14602 raft_consensus.cc:3060] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.374219 15699 raft_consensus.cc:2804] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:50.375705 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cbd18e927c843e7b0aee7e0ccb1b06d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:50.376168 14600 raft_consensus.cc:3060] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.386492 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ab48f6288494005ad42208e2e97ddf2" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:50.387115 14606 raft_consensus.cc:3060] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.387188 15699 raft_consensus.cc:697] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:50.387897 15771 raft_consensus.cc:2749] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:50.388113 15699 consensus_queue.cc:237] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.390177 14714 consensus_queue.cc:260] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.392019 14714 ts_tablet_manager.cc:1434] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.034s user 0.001s sys 0.003s
I20251211 08:57:50.394898 15648 raft_consensus.cc:493] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:50.395382 15648 raft_consensus.cc:515] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.397739 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98d14ba15dcd47f8b9e8035b3c7d477c" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:50.398229 14296 raft_consensus.cc:2468] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:50.398860 14602 raft_consensus.cc:2468] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:50.398578 14606 raft_consensus.cc:2468] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:50.399327 14385 leader_election.cc:304] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:50.400040 14382 leader_election.cc:304] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:50.400444 15756 raft_consensus.cc:2804] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251211 08:57:50.400663 14382 raft_consensus.cc:2686] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251211 08:57:50.400821 15756 raft_consensus.cc:493] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:50.401188 15756 raft_consensus.cc:3060] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.401358 14382 leader_election.cc:304] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:50.402107 15696 raft_consensus.cc:2804] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:50.402570 15696 raft_consensus.cc:697] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:50.402921 15648 leader_election.cc:290] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.402913 14905 rpcz_store.cc:276] 1211 08:57:44.441717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.441835 (+ 118us) service_pool.cc:225] Handling call
1211 08:57:50.294245 (+5852410us) ts_tablet_manager.cc:1151] Deleting tablet 5d8d84e594ea47f3bc9127cc587e8aa7
1211 08:57:50.294282 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.294507 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5851917}
I20251211 08:57:50.360122 14608 raft_consensus.cc:2468] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:50.252691 15514 raft_consensus.cc:515] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.403278 15696 consensus_queue.cc:237] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.405004 15514 leader_election.cc:290] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:50.408912 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f785a3cc2e8145eebae7e9c2885b0b1e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:50.409981 14542 leader_election.cc:343] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.411936 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 380) took 2042 ms. Trace:
I20251211 08:57:50.413950 15756 raft_consensus.cc:515] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.415918 14600 raft_consensus.cc:2468] T 4cbd18e927c843e7b0aee7e0ccb1b06d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
W20251211 08:57:50.418295 14004 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 814f31ce84d648ca8104513c7d6c39ae has no consensus state
I20251211 08:57:50.415406 14714 tablet_bootstrap.cc:492] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:50.419905 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet aadc3c1ecd974ae3a0d491bd209047e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aadc3c1ecd974ae3a0d491bd209047e6 already in progress: creating tablet
I20251211 08:57:50.423921 15756 leader_election.cc:290] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.426785 15751 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251211 08:57:50.417487 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98d14ba15dcd47f8b9e8035b3c7d477c" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:50.427824 14296 raft_consensus.cc:3060] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.427696 14904 rpcz_store.cc:276] 1211 08:57:48.369454 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.369555 (+ 101us) service_pool.cc:225] Handling call
1211 08:57:50.411586 (+2042031us) ts_tablet_manager.cc:1151] Deleting tablet f3cf2b4d14ba45d79e52b6fdb0497d58
1211 08:57:50.411634 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.411882 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2041577}
W20251211 08:57:50.431149 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet f3cf2b4d14ba45d79e52b6fdb0497d58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3cf2b4d14ba45d79e52b6fdb0497d58 already in progress: creating tablet
I20251211 08:57:50.435426 14296 raft_consensus.cc:2468] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:50.436480 14385 leader_election.cc:304] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
W20251211 08:57:50.440173 14542 leader_election.cc:343] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 3c2fb2d9564c43ee84b8dfcea8ec54b0
I20251211 08:57:50.440604 15739 leader_election.cc:290] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:50.441558 15696 raft_consensus.cc:2804] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:50.442107 15696 raft_consensus.cc:697] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:50.435647 15774 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251211 08:57:50.442968 15696 consensus_queue.cc:237] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.436978 15030 tablet_bootstrap.cc:492] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:50.444082 15030 ts_tablet_manager.cc:1403] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.503s user 0.009s sys 0.015s
I20251211 08:57:50.442435 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 381) took 2019 ms. Trace:
I20251211 08:57:50.448398 14904 rpcz_store.cc:276] 1211 08:57:48.422658 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.454146 (+ 31488us) service_pool.cc:225] Handling call
1211 08:57:50.442072 (+1987926us) ts_tablet_manager.cc:1151] Deleting tablet aadc3c1ecd974ae3a0d491bd209047e6
1211 08:57:50.442112 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.442384 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1987451}
W20251211 08:57:50.449621 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet aadc3c1ecd974ae3a0d491bd209047e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aadc3c1ecd974ae3a0d491bd209047e6 already in progress: creating tablet
I20251211 08:57:50.453625 15030 raft_consensus.cc:359] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.454411 15030 raft_consensus.cc:385] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:50.454705 15030 raft_consensus.cc:740] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
W20251211 08:57:50.456434 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 83471us
I20251211 08:57:50.457458 14230 leader_election.cc:304] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
W20251211 08:57:50.469604 14382 leader_election.cc:343] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 98d14ba15dcd47f8b9e8035b3c7d477c
I20251211 08:57:50.472688 14714 tablet_bootstrap.cc:654] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:50.474637 15778 tablet.cc:1722] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251211 08:57:50.483438 14382 leader_election.cc:343] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 98d14ba15dcd47f8b9e8035b3c7d477c
W20251211 08:57:50.497507 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 165000us
I20251211 08:57:50.499759 14905 tablet_replica.cc:333] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:50.500590 14905 raft_consensus.cc:2243] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:50.501068 14905 raft_consensus.cc:2272] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:50.500366 15030 consensus_queue.cc:260] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.518064 15299 ts_tablet_manager.cc:1929] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:50.518440 15299 log.cc:1199] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/c6d0fef50524476dbd47fc70e4bda319
I20251211 08:57:50.519079 15299 ts_tablet_manager.cc:1950] T c6d0fef50524476dbd47fc70e4bda319 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:50.521031 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 417) took 2009 ms. Trace:
I20251211 08:57:50.521368 15299 rpcz_store.cc:276] 1211 08:57:48.511723 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.516757 (+ 5034us) service_pool.cc:225] Handling call
1211 08:57:50.349226 (+1832469us) ts_tablet_manager.cc:1151] Deleting tablet c6d0fef50524476dbd47fc70e4bda319
1211 08:57:50.349285 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.425423 (+ 76138us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.517827 (+ 92404us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.520929 (+ 3102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1832016}
I20251211 08:57:50.522660 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet c6d0fef50524476dbd47fc70e4bda319 (table default.Table_7919ce70cfc5400bb4f2515164c6b7a1 [id=3a336d5ad7004619951e901ad9bf39bf]) successfully deleted
I20251211 08:57:50.524127 15770 raft_consensus.cc:2804] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:50.524653 15770 raft_consensus.cc:697] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:50.525513 15770 consensus_queue.cc:237] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.528674 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_034c882ef42046d58a86c1b90f5cdf85" } modify_external_catalogs: true
I20251211 08:57:50.529103 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_034c882ef42046d58a86c1b90f5cdf85" } modify_external_catalogs: true
W20251211 08:57:50.535387 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 78017.2us
I20251211 08:57:50.522526 15299 tablet_replica.cc:333] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:50.481297 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 382) took 2025 ms. Trace:
I20251211 08:57:50.537324 14904 rpcz_store.cc:276] 1211 08:57:48.455495 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.455748 (+ 253us) service_pool.cc:225] Handling call
1211 08:57:50.480994 (+2025246us) ts_tablet_manager.cc:1151] Deleting tablet c6d0fef50524476dbd47fc70e4bda319
1211 08:57:50.481032 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.481238 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2024821}
I20251211 08:57:50.538049 14714 tablet_bootstrap.cc:492] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
W20251211 08:57:50.538473 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet c6d0fef50524476dbd47fc70e4bda319 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6d0fef50524476dbd47fc70e4bda319 already in progress: creating tablet
I20251211 08:57:50.538800 14904 tablet_replica.cc:333] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:50.482411 14722 tablet_bootstrap.cc:492] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:50.539084 14714 ts_tablet_manager.cc:1403] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.124s user 0.010s sys 0.000s
I20251211 08:57:50.539464 14722 ts_tablet_manager.cc:1403] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.289s user 0.010s sys 0.000s
I20251211 08:57:50.541482 14714 raft_consensus.cc:359] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.541496 14722 raft_consensus.cc:359] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.542057 14714 raft_consensus.cc:385] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:50.542275 14714 raft_consensus.cc:740] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:50.542323 14722 raft_consensus.cc:385] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:50.542680 14722 raft_consensus.cc:740] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:50.542924 14714 consensus_queue.cc:260] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.553205 14714 ts_tablet_manager.cc:1434] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20251211 08:57:50.554009 15299 raft_consensus.cc:2243] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:50.554699 15299 pending_rounds.cc:70] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:50.555003 15299 pending_rounds.cc:77] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256453837553664 op_type: NO_OP noop_request { }
I20251211 08:57:50.555423 15299 raft_consensus.cc:2889] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:50.555732 15299 raft_consensus.cc:2272] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:50.565737 14714 tablet_bootstrap.cc:492] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:50.566722 15030 ts_tablet_manager.cc:1434] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.122s user 0.006s sys 0.000s
I20251211 08:57:50.567662 15030 tablet_bootstrap.cc:492] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:50.502733 14905 ts_tablet_manager.cc:1916] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:50.583403 15299 ts_tablet_manager.cc:1916] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:50.587390 14714 tablet_bootstrap.cc:654] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:50.600714 14722 consensus_queue.cc:260] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.549151 14904 raft_consensus.cc:2243] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:50.557531 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f785a3cc2e8145eebae7e9c2885b0b1e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:50.570945 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c2fb2d9564c43ee84b8dfcea8ec54b0" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:50.603180 14904 raft_consensus.cc:2272] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:50.602725 15743 raft_consensus.cc:493] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:50.605446 15743 raft_consensus.cc:515] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.605942 14904 ts_tablet_manager.cc:1916] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:50.608521 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f7775994f684326b6312c802f6b084d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:50.609143 14447 raft_consensus.cc:2468] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:50.610193 14231 leader_election.cc:304] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:50.610834 15699 raft_consensus.cc:2804] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:50.611124 15699 raft_consensus.cc:493] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:50.611379 15699 raft_consensus.cc:3060] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.609284 14714 tablet_bootstrap.cc:492] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:50.621079 14714 ts_tablet_manager.cc:1403] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.000s
I20251211 08:57:50.621475 15699 raft_consensus.cc:515] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.623611 14714 raft_consensus.cc:359] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.623960 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f7775994f684326b6312c802f6b084d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:50.624316 14714 raft_consensus.cc:385] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:50.624467 14447 raft_consensus.cc:3060] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.624586 14714 raft_consensus.cc:740] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:50.625247 14714 consensus_queue.cc:260] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.632683 14714 ts_tablet_manager.cc:1434] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20251211 08:57:50.635366 14447 raft_consensus.cc:2468] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:50.635846 15299 ts_tablet_manager.cc:1929] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:50.636363 14231 leader_election.cc:304] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:50.637214 15770 raft_consensus.cc:2804] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:50.637652 15770 raft_consensus.cc:697] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:50.639650 15299 log.cc:1199] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/da57b72207414512b9744b283bc869fe
I20251211 08:57:50.638697 15770 consensus_queue.cc:237] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.642802 14722 ts_tablet_manager.cc:1434] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.103s user 0.005s sys 0.000s
I20251211 08:57:50.644709 15743 leader_election.cc:290] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.643950 14722 tablet_bootstrap.cc:492] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:50.645718 14544 leader_election.cc:343] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.646106 14544 leader_election.cc:304] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:50.646987 15739 raft_consensus.cc:2749] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:50.647997 14544 leader_election.cc:343] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.637287 14714 tablet_bootstrap.cc:492] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:50.648304 14544 leader_election.cc:304] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:50.650391 15739 raft_consensus.cc:2749] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251211 08:57:50.651291 15699 leader_election.cc:290] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.652945 14714 tablet_bootstrap.cc:654] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:50.655122 15299 ts_tablet_manager.cc:1950] T da57b72207414512b9744b283bc869fe P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
W20251211 08:57:50.655083 14230 leader_election.cc:343] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 5f7775994f684326b6312c802f6b084d
W20251211 08:57:50.655817 14230 leader_election.cc:343] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 5f7775994f684326b6312c802f6b084d
I20251211 08:57:50.658020 15030 tablet_bootstrap.cc:654] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:50.672205 14714 tablet_bootstrap.cc:492] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:50.672639 14714 ts_tablet_manager.cc:1403] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.001s
I20251211 08:57:50.673620 15739 raft_consensus.cc:493] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:50.674100 15739 raft_consensus.cc:515] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.674980 14714 raft_consensus.cc:359] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.675588 14714 raft_consensus.cc:385] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:50.675806 14714 raft_consensus.cc:740] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:50.676949 15739 leader_election.cc:290] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:50.677613 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07aef1eea7c04d9cbf35c82e45df7a16" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:50.678653 14542 leader_election.cc:343] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.679751 14011 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b58bca5097ae4e3cab8c112bc4d17bbb" } new_table_name: "default.Table_000d74366d2342a09dc91f19a5ccdc9f" modify_external_catalogs: true
I20251211 08:57:50.687042 14714 consensus_queue.cc:260] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.689496 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07aef1eea7c04d9cbf35c82e45df7a16" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:50.691694 13999 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 814f31ce84d648ca8104513c7d6c39ae (table default.Table_60479fec7ce247e687a133fd0dbad564 [id=cd2cef42eefc41bfbc8d7cdcad060d69], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 814f31ce84d648ca8104513c7d6c39ae has no consensus state
I20251211 08:57:50.692538 14714 ts_tablet_manager.cc:1434] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.020s user 0.002s sys 0.004s
W20251211 08:57:50.697324 14544 leader_election.cc:343] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.697644 14544 leader_election.cc:304] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:50.698632 15739 raft_consensus.cc:2749] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:50.700349 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 419) took 1921 ms. Trace:
I20251211 08:57:50.700589 15299 rpcz_store.cc:276] 1211 08:57:48.778587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.778743 (+ 156us) service_pool.cc:225] Handling call
1211 08:57:50.522354 (+1743611us) ts_tablet_manager.cc:1151] Deleting tablet da57b72207414512b9744b283bc869fe
1211 08:57:50.522410 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.607268 (+ 84858us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.635632 (+ 28364us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.700282 (+ 64650us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1689141}
I20251211 08:57:50.701634 15299 tablet_replica.cc:333] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:50.702337 15299 raft_consensus.cc:2243] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:50.702756 14009 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 4b9ae463731d4828afc02039e0d161b5 (table default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]). Expected version 1 got 0
I20251211 08:57:50.703425 14714 tablet_bootstrap.cc:492] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:50.704408 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet da57b72207414512b9744b283bc869fe (table default.Table_2f325f51dd9f433f98cfef7169cdbe0c [id=2ff95cefc4a545398f979e2bd0321f5a]) successfully deleted
I20251211 08:57:50.707304 14421 tablet_service.cc:1505] Processing CreateTablet for tablet da428d3148274814ad469415354be137 (DEFAULT_TABLE table=default.Table_0c957f35d448462b9e819895f4832873 [id=7eb5335814b748a395ee6294ca153bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.708514 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da428d3148274814ad469415354be137. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:50.705349 14009 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 89) took 1178 ms (client timeout 1000 ms). Trace:
I20251211 08:57:50.709324 14580 tablet_service.cc:1505] Processing CreateTablet for tablet da428d3148274814ad469415354be137 (DEFAULT_TABLE table=default.Table_0c957f35d448462b9e819895f4832873 [id=7eb5335814b748a395ee6294ca153bd8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:50.709344 14009 rpcz_store.cc:269] 1211 08:57:49.527241 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.527329 (+ 88us) service_pool.cc:225] Handling call
1211 08:57:49.551308 (+ 23979us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:50.705275 (+1153967us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":51126528}
I20251211 08:57:50.706521 14005 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
I20251211 08:57:50.711318 14416 tablet_service.cc:1505] Processing CreateTablet for tablet 58ec593a015d469e991473591a2a7bad (DEFAULT_TABLE table=default.Table_e0dbfa29ddc84b3e87f44bc9ceb301a6 [id=0db2a12e10a547838963be213c2fa546]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.712406 14416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ec593a015d469e991473591a2a7bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.714114 15696 raft_consensus.cc:493] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:50.714581 15696 raft_consensus.cc:515] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.715286 14004 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
I20251211 08:57:50.716574 15696 leader_election.cc:290] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.716727 14278 tablet_service.cc:1505] Processing CreateTablet for tablet da428d3148274814ad469415354be137 (DEFAULT_TABLE table=default.Table_0c957f35d448462b9e819895f4832873 [id=7eb5335814b748a395ee6294ca153bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.717818 14271 tablet_service.cc:1505] Processing CreateTablet for tablet 58ec593a015d469e991473591a2a7bad (DEFAULT_TABLE table=default.Table_e0dbfa29ddc84b3e87f44bc9ceb301a6 [id=0db2a12e10a547838963be213c2fa546]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.718904 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 51599b4a82644febb80c103f2281a3cc (DEFAULT_TABLE table=default.Table_f526612840624a878073018b98efcd87 [id=adb6d725ab054586a6c4437521f5d1a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.719954 14265 tablet_service.cc:1505] Processing CreateTablet for tablet 814f31ce84d648ca8104513c7d6c39ae (DEFAULT_TABLE table=default.Table_60479fec7ce247e687a133fd0dbad564 [id=cd2cef42eefc41bfbc8d7cdcad060d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.721123 14013 catalog_manager.cc:5654] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:50.722302 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 4b9ae463731d4828afc02039e0d161b5 (table default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]). Expected version 1 got 0
I20251211 08:57:50.724138 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b9ae463731d4828afc02039e0d161b5" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:50.726555 14276 tablet_service.cc:1505] Processing CreateTablet for tablet abc0719b595c407c91811a507e6e20db (DEFAULT_TABLE table=default.Table_05b7dc5c661e4002b1fee6521138d5a5 [id=c8f9fbd9972b43f18b3e92607877acef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.727581 15299 pending_rounds.cc:70] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:50.727789 15299 pending_rounds.cc:77] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256453691162624 op_type: NO_OP noop_request { }
I20251211 08:57:50.727804 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b9ae463731d4828afc02039e0d161b5" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:50.727774 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 58ec593a015d469e991473591a2a7bad (DEFAULT_TABLE table=default.Table_e0dbfa29ddc84b3e87f44bc9ceb301a6 [id=0db2a12e10a547838963be213c2fa546]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.728103 15299 raft_consensus.cc:2889] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:50.728387 15299 raft_consensus.cc:2272] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:50.728516 14600 raft_consensus.cc:2393] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
W20251211 08:57:50.713061 14005 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 54) took 878 ms (client timeout 1000 ms). Trace:
W20251211 08:57:50.729251 14005 rpcz_store.cc:269] 1211 08:57:49.834665 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.837256 (+ 2591us) service_pool.cc:225] Handling call
1211 08:57:50.713011 (+875755us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3}
I20251211 08:57:50.730136 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 51599b4a82644febb80c103f2281a3cc (DEFAULT_TABLE table=default.Table_f526612840624a878073018b98efcd87 [id=adb6d725ab054586a6c4437521f5d1a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.731842 14382 leader_election.cc:304] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:50.732175 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 814f31ce84d648ca8104513c7d6c39ae (DEFAULT_TABLE table=default.Table_60479fec7ce247e687a133fd0dbad564 [id=cd2cef42eefc41bfbc8d7cdcad060d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.732187 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 7a2ccf777dbd41cd807d01b395b0aaa1 (DEFAULT_TABLE table=default.Table_a15d62e9febf43b3a22ef81fb3feafb6 [id=761730dd01bc41bda881b0e0c183fbb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.732897 14423 tablet_service.cc:1505] Processing CreateTablet for tablet 814f31ce84d648ca8104513c7d6c39ae (DEFAULT_TABLE table=default.Table_60479fec7ce247e687a133fd0dbad564 [id=cd2cef42eefc41bfbc8d7cdcad060d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.730499 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet ccce5a446ac9460c9c9b7de72d875d95
I20251211 08:57:50.733229 15696 raft_consensus.cc:3060] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.733943 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 814f31ce84d648ca8104513c7d6c39ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.734493 14012 catalog_manager.cc:6724] ReplaceTablet: tablet ccce5a446ac9460c9c9b7de72d875d95 deleted and replaced by tablet 1ce1ef2236f54feab5c38bc401c126e8
I20251211 08:57:50.735667 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 51599b4a82644febb80c103f2281a3cc (DEFAULT_TABLE table=default.Table_f526612840624a878073018b98efcd87 [id=adb6d725ab054586a6c4437521f5d1a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.736428 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet ccce5a446ac9460c9c9b7de72d875d95 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ce1ef2236f54feab5c38bc401c126e8) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:50.736716 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51599b4a82644febb80c103f2281a3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.737373 14420 tablet_service.cc:1505] Processing CreateTablet for tablet abc0719b595c407c91811a507e6e20db (DEFAULT_TABLE table=default.Table_05b7dc5c661e4002b1fee6521138d5a5 [id=c8f9fbd9972b43f18b3e92607877acef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.737818 14714 tablet_bootstrap.cc:654] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:50.738241 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc0719b595c407c91811a507e6e20db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.739444 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet ccce5a446ac9460c9c9b7de72d875d95 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ce1ef2236f54feab5c38bc401c126e8) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:50.741407 15299 ts_tablet_manager.cc:1916] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:50.742977 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
I20251211 08:57:50.743698 14013 catalog_manager.cc:5654] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:50.749657 14586 tablet_service.cc:1505] Processing CreateTablet for tablet abc0719b595c407c91811a507e6e20db (DEFAULT_TABLE table=default.Table_05b7dc5c661e4002b1fee6521138d5a5 [id=c8f9fbd9972b43f18b3e92607877acef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.752077 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet ccce5a446ac9460c9c9b7de72d875d95 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ce1ef2236f54feab5c38bc401c126e8) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:50.754355 15696 raft_consensus.cc:2749] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:50.769678 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 7a2ccf777dbd41cd807d01b395b0aaa1 (DEFAULT_TABLE table=default.Table_a15d62e9febf43b3a22ef81fb3feafb6 [id=761730dd01bc41bda881b0e0c183fbb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.770716 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 7a2ccf777dbd41cd807d01b395b0aaa1 (DEFAULT_TABLE table=default.Table_a15d62e9febf43b3a22ef81fb3feafb6 [id=761730dd01bc41bda881b0e0c183fbb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.772807 14714 tablet_bootstrap.cc:492] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:50.773193 14714 ts_tablet_manager.cc:1403] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.005s
I20251211 08:57:50.775429 14714 raft_consensus.cc:359] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.775971 14714 raft_consensus.cc:385] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:50.776202 14714 raft_consensus.cc:740] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:50.778072 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 827b6a73ea5a4462ae58f36f12919b0c (DEFAULT_TABLE table=default.Table_4d9c098623e84435972e3c148a08b83e [id=e777b0c9fe014c3f94da539cdf1196a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.778725 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_41112ce230a642ddb3f81bb1ddbb7916"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:50.780381 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41112ce230a642ddb3f81bb1ddbb7916 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:50.782478 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_24bf44a5acdf49b3803c8dc014433797"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:50.784060 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24bf44a5acdf49b3803c8dc014433797 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:50.779048 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_e6479b81db9f43c9a99abea4806decd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:50.787302 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6479b81db9f43c9a99abea4806decd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:50.788424 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a2ccf777dbd41cd807d01b395b0aaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.801000 15696 raft_consensus.cc:493] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:50.801544 15696 raft_consensus.cc:515] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.803712 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 856db6122fab4e54bbd9f22f24363823 (DEFAULT_TABLE table=default.Table_1c1540b4735a45c39b48fe0c26d9b89b [id=f2a38de0f2d5404b9d7bb922760d5fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.807082 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da428d3148274814ad469415354be137. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.808851 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 827b6a73ea5a4462ae58f36f12919b0c (DEFAULT_TABLE table=default.Table_4d9c098623e84435972e3c148a08b83e [id=e777b0c9fe014c3f94da539cdf1196a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.809713 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827b6a73ea5a4462ae58f36f12919b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.811208 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ec593a015d469e991473591a2a7bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.810931 15792 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251211 08:57:50.811676 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51599b4a82644febb80c103f2281a3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.815248 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 814f31ce84d648ca8104513c7d6c39ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.819182 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc0719b595c407c91811a507e6e20db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.823287 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a2ccf777dbd41cd807d01b395b0aaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.828789 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 63cb7025541d4793aa6f8d0e63689544 (DEFAULT_TABLE table=default.Table_3b53cda0501e4b3fa6c51534dc6fa05e [id=50d950f6c04e46e29c56d0b51e598cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.829281 14576 tablet_service.cc:1505] Processing CreateTablet for tablet 856db6122fab4e54bbd9f22f24363823 (DEFAULT_TABLE table=default.Table_1c1540b4735a45c39b48fe0c26d9b89b [id=f2a38de0f2d5404b9d7bb922760d5fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.830135 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856db6122fab4e54bbd9f22f24363823. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.835762 14269 tablet_service.cc:1505] Processing CreateTablet for tablet 2f9c298049734abab7bd934ecf208353 (DEFAULT_TABLE table=default.Table_952b121ba0144ec3975634134ac3c07d [id=e32cfdeeb8df498db6473b5a81652f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.836910 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da52cace2aab4f9dacef18ec5fa56aaf" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:50.838006 14382 leader_election.cc:343] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.838457 15696 leader_election.cc:290] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.838500 15299 ts_tablet_manager.cc:1929] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:50.839594 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da52cace2aab4f9dacef18ec5fa56aaf" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:50.840173 14299 raft_consensus.cc:2468] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:50.847693 15299 log.cc:1199] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/49015c5d4f2b4d07b547a9df57c992eb
I20251211 08:57:50.841238 14385 leader_election.cc:304] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:50.848587 15299 ts_tablet_manager.cc:1950] T 49015c5d4f2b4d07b547a9df57c992eb P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:50.849145 15696 raft_consensus.cc:2804] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:50.847236 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 827b6a73ea5a4462ae58f36f12919b0c (DEFAULT_TABLE table=default.Table_4d9c098623e84435972e3c148a08b83e [id=e777b0c9fe014c3f94da539cdf1196a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.849467 15696 raft_consensus.cc:493] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:50.810473 14714 consensus_queue.cc:260] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.849843 15696 raft_consensus.cc:3060] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.844650 14421 tablet_service.cc:1505] Processing CreateTablet for tablet 63cb7025541d4793aa6f8d0e63689544 (DEFAULT_TABLE table=default.Table_3b53cda0501e4b3fa6c51534dc6fa05e [id=50d950f6c04e46e29c56d0b51e598cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.853631 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 63cb7025541d4793aa6f8d0e63689544 (DEFAULT_TABLE table=default.Table_3b53cda0501e4b3fa6c51534dc6fa05e [id=50d950f6c04e46e29c56d0b51e598cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.855479 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63cb7025541d4793aa6f8d0e63689544. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.854547 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827b6a73ea5a4462ae58f36f12919b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.856264 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63cb7025541d4793aa6f8d0e63689544. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.858487 14905 ts_tablet_manager.cc:1929] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:50.862648 14714 ts_tablet_manager.cc:1434] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.089s user 0.004s sys 0.003s
I20251211 08:57:50.863101 14416 tablet_service.cc:1505] Processing CreateTablet for tablet 2f9c298049734abab7bd934ecf208353 (DEFAULT_TABLE table=default.Table_952b121ba0144ec3975634134ac3c07d [id=e32cfdeeb8df498db6473b5a81652f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.878049 15696 raft_consensus.cc:515] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.880311 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 422) took 1991 ms. Trace:
I20251211 08:57:50.882850 14268 tablet_service.cc:1505] Processing CreateTablet for tablet db8992d01d8544d5960bb595877e6da4 (DEFAULT_TABLE table=default.Table_91192655bac749ae8e0649eb05ab6179 [id=327d7ff89d264982b1b47c407c89d80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.886966 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 5f8b9863a7fe4e8496e42b73d71066a8 (DEFAULT_TABLE table=default.Table_9776c6b19dd141b9ac63ce2c700c2a0e [id=f4f64c66e8894d9e8b17e5e7681d128f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.888291 14420 tablet_service.cc:1505] Processing CreateTablet for tablet db8992d01d8544d5960bb595877e6da4 (DEFAULT_TABLE table=default.Table_91192655bac749ae8e0649eb05ab6179 [id=327d7ff89d264982b1b47c407c89d80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.888944 15299 rpcz_store.cc:276] 1211 08:57:48.888814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.889254 (+ 440us) service_pool.cc:225] Handling call
1211 08:57:50.701489 (+1812235us) ts_tablet_manager.cc:1151] Deleting tablet 49015c5d4f2b4d07b547a9df57c992eb
1211 08:57:50.701536 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.776911 (+ 75375us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.838279 (+ 61368us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.880254 (+ 41975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1634914}
I20251211 08:57:50.871460 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 2f9c298049734abab7bd934ecf208353 (DEFAULT_TABLE table=default.Table_952b121ba0144ec3975634134ac3c07d [id=e32cfdeeb8df498db6473b5a81652f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.895982 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9c298049734abab7bd934ecf208353. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.899627 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 49015c5d4f2b4d07b547a9df57c992eb (table default.Table_73091d56834e4e78b870257988c87c57 [id=d8f07591554543f2880445cb636da15a]) successfully deleted
W20251211 08:57:50.901258 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 225438e70aae424a9671aaceb3ee0ed6 has no consensus state
I20251211 08:57:50.885927 14580 tablet_service.cc:1505] Processing CreateTablet for tablet db8992d01d8544d5960bb595877e6da4 (DEFAULT_TABLE table=default.Table_91192655bac749ae8e0649eb05ab6179 [id=327d7ff89d264982b1b47c407c89d80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.902303 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da52cace2aab4f9dacef18ec5fa56aaf" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:50.902841 14299 raft_consensus.cc:3060] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:50.903168 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db8992d01d8544d5960bb595877e6da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.909461 14416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9c298049734abab7bd934ecf208353. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.909653 14714 tablet_bootstrap.cc:492] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:50.909893 14299 raft_consensus.cc:2468] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:50.910787 14385 leader_election.cc:304] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:50.911464 15650 raft_consensus.cc:2804] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:50.911866 15650 raft_consensus.cc:697] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:50.912561 15650 consensus_queue.cc:237] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:50.914338 15756 consensus_queue.cc:1048] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:50.915506 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db8992d01d8544d5960bb595877e6da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.916416 14714 tablet_bootstrap.cc:654] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:50.916635 14385 consensus_peers.cc:597] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:50.917603 14423 tablet_service.cc:1505] Processing CreateTablet for tablet 5f8b9863a7fe4e8496e42b73d71066a8 (DEFAULT_TABLE table=default.Table_9776c6b19dd141b9ac63ce2c700c2a0e [id=f4f64c66e8894d9e8b17e5e7681d128f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.918458 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f8b9863a7fe4e8496e42b73d71066a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.918957 15299 tablet_replica.cc:333] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:50.919678 15299 raft_consensus.cc:2243] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:50.920284 15299 pending_rounds.cc:70] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:50.920477 15299 pending_rounds.cc:77] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256454781530112 op_type: NO_OP noop_request { }
I20251211 08:57:50.920794 15299 raft_consensus.cc:2889] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:50.921105 15299 raft_consensus.cc:2272] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:50.923286 15299 ts_tablet_manager.cc:1916] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:50.931998 14905 log.cc:1199] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/f5b462cf21e3420c8e323a4ac4f04abe
I20251211 08:57:50.932714 14905 ts_tablet_manager.cc:1950] T f5b462cf21e3420c8e323a4ac4f04abe P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:50.940713 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 330) took 6370 ms. Trace:
I20251211 08:57:50.941150 14905 rpcz_store.cc:276] 1211 08:57:44.570495 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.571095 (+ 600us) service_pool.cc:225] Handling call
1211 08:57:50.407120 (+5836025us) ts_tablet_manager.cc:1151] Deleting tablet f5b462cf21e3420c8e323a4ac4f04abe
1211 08:57:50.461201 (+ 54081us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.774956 (+313755us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.858304 (+ 83348us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.940660 (+ 82356us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5835536}
I20251211 08:57:50.942550 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet f5b462cf21e3420c8e323a4ac4f04abe (table default.Table_e444843f69064fbb9c2a6d0671fd9a2a [id=6c43f429406443d7b5834905929dcbd4]) successfully deleted
I20251211 08:57:50.944247 15696 leader_election.cc:290] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:50.945214 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da52cace2aab4f9dacef18ec5fa56aaf" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
W20251211 08:57:50.946489 14382 leader_election.cc:343] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:50.947367 14587 tablet_service.cc:1505] Processing CreateTablet for tablet 5f8b9863a7fe4e8496e42b73d71066a8 (DEFAULT_TABLE table=default.Table_9776c6b19dd141b9ac63ce2c700c2a0e [id=f4f64c66e8894d9e8b17e5e7681d128f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.948287 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f8b9863a7fe4e8496e42b73d71066a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.948845 14272 tablet_service.cc:1505] Processing CreateTablet for tablet 8298fc5a055148be8c1b67ed936a8289 (DEFAULT_TABLE table=default.Table_2d03f7f268c3461ca7e972ca69273fef [id=bdc7ab57edbb4c2f914d0a247505ffbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.963152 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 856db6122fab4e54bbd9f22f24363823 (DEFAULT_TABLE table=default.Table_1c1540b4735a45c39b48fe0c26d9b89b [id=f2a38de0f2d5404b9d7bb922760d5fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.973587 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 8298fc5a055148be8c1b67ed936a8289 (DEFAULT_TABLE table=default.Table_2d03f7f268c3461ca7e972ca69273fef [id=bdc7ab57edbb4c2f914d0a247505ffbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.974574 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8298fc5a055148be8c1b67ed936a8289. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.974925 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 4849a22585f24e388e3d8d6f4277c997 (DEFAULT_TABLE table=default.Table_d3f2e5059297427fbaaefd79504a2e2f [id=945a9eb17575451192d09cf17a00a46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.972618 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 8298fc5a055148be8c1b67ed936a8289 (DEFAULT_TABLE table=default.Table_2d03f7f268c3461ca7e972ca69273fef [id=bdc7ab57edbb4c2f914d0a247505ffbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.986603 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8298fc5a055148be8c1b67ed936a8289. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.673272 14722 tablet_bootstrap.cc:654] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:50.991196 14421 tablet_service.cc:1505] Processing CreateTablet for tablet 4849a22585f24e388e3d8d6f4277c997 (DEFAULT_TABLE table=default.Table_d3f2e5059297427fbaaefd79504a2e2f [id=945a9eb17575451192d09cf17a00a46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:50.992233 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4849a22585f24e388e3d8d6f4277c997. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.997013 15030 tablet_bootstrap.cc:492] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:50.997417 15030 ts_tablet_manager.cc:1403] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.430s user 0.007s sys 0.003s
I20251211 08:57:50.998325 15299 ts_tablet_manager.cc:1929] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:50.998660 15299 log.cc:1199] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/4ab48f6288494005ad42208e2e97ddf2
I20251211 08:57:50.999393 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856db6122fab4e54bbd9f22f24363823. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.999356 15299 ts_tablet_manager.cc:1950] T 4ab48f6288494005ad42208e2e97ddf2 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:51.000869 15030 raft_consensus.cc:359] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.001573 15030 raft_consensus.cc:385] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:51.001888 15030 raft_consensus.cc:740] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:51.002658 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 428) took 1787 ms. Trace:
I20251211 08:57:51.002954 15299 rpcz_store.cc:276] 1211 08:57:49.215375 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.215465 (+ 90us) service_pool.cc:225] Handling call
1211 08:57:50.889922 (+1674457us) ts_tablet_manager.cc:1151] Deleting tablet 4ab48f6288494005ad42208e2e97ddf2
1211 08:57:50.918848 (+ 28926us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.953144 (+ 34296us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:50.997536 (+ 44392us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:51.002609 (+ 5073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673953}
I20251211 08:57:51.003986 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 4ab48f6288494005ad42208e2e97ddf2 (table default.Table_cd9ec56dc08c48779e88a778d2917bb0 [id=658ce49773af41519ab660f3bc29975b]) successfully deleted
I20251211 08:57:51.007820 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 430) took 1791 ms. Trace:
I20251211 08:57:51.008862 15299 rpcz_store.cc:276] 1211 08:57:49.215924 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.218848 (+ 2924us) service_pool.cc:225] Handling call
1211 08:57:51.007488 (+1788640us) ts_tablet_manager.cc:1151] Deleting tablet f2f095b59ecf44ffabaa0ac22212938b
1211 08:57:51.007539 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:51.007776 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1788165}
W20251211 08:57:51.010475 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f2f095b59ecf44ffabaa0ac22212938b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2f095b59ecf44ffabaa0ac22212938b already in progress: creating tablet
I20251211 08:57:51.011358 15299 tablet_replica.cc:333] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:51.012542 15299 raft_consensus.cc:2243] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:51.015174 15299 raft_consensus.cc:2272] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:51.018208 15299 ts_tablet_manager.cc:1916] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:51.027428 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da428d3148274814ad469415354be137. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.027897 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51599b4a82644febb80c103f2281a3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.028368 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a2ccf777dbd41cd807d01b395b0aaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.029098 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9c298049734abab7bd934ecf208353. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.030062 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc0719b595c407c91811a507e6e20db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.033109 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827b6a73ea5a4462ae58f36f12919b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.034986 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ec593a015d469e991473591a2a7bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.036898 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63cb7025541d4793aa6f8d0e63689544. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.038995 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 814f31ce84d648ca8104513c7d6c39ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.041841 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856db6122fab4e54bbd9f22f24363823. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.047081 15650 raft_consensus.cc:493] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:51.047572 15650 raft_consensus.cc:515] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.048799 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 342) took 6141 ms. Trace:
I20251211 08:57:51.049113 14905 rpcz_store.cc:276] 1211 08:57:44.907753 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.910909 (+ 3156us) service_pool.cc:225] Handling call
1211 08:57:51.048417 (+6137508us) ts_tablet_manager.cc:1151] Deleting tablet 4631dd0bc0c144e69a31de80de39a76a
1211 08:57:51.048463 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:51.048706 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6136888}
I20251211 08:57:51.049425 15650 leader_election.cc:290] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:51.050256 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 4631dd0bc0c144e69a31de80de39a76a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4631dd0bc0c144e69a31de80de39a76a already in progress: creating tablet
I20251211 08:57:51.051007 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd14de369e084828afb8a924c46c649b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:51.051209 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db8992d01d8544d5960bb595877e6da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.052452 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4849a22585f24e388e3d8d6f4277c997. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:50.985762 14714 tablet_bootstrap.cc:492] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:51.053901 14714 ts_tablet_manager.cc:1403] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.144s user 0.010s sys 0.000s
I20251211 08:57:51.054983 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f8b9863a7fe4e8496e42b73d71066a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.056324 14714 raft_consensus.cc:359] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.056949 14714 raft_consensus.cc:385] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:51.057184 14714 raft_consensus.cc:740] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
W20251211 08:57:51.059201 14385 leader_election.cc:343] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:51.059930 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8298fc5a055148be8c1b67ed936a8289. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:51.075685 14382 leader_election.cc:343] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: dd14de369e084828afb8a924c46c649b
I20251211 08:57:51.076069 14382 leader_election.cc:304] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:51.076977 15650 raft_consensus.cc:2749] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:51.080442 14579 tablet_service.cc:1505] Processing CreateTablet for tablet 4849a22585f24e388e3d8d6f4277c997 (DEFAULT_TABLE table=default.Table_d3f2e5059297427fbaaefd79504a2e2f [id=945a9eb17575451192d09cf17a00a46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.081353 14263 tablet_service.cc:1505] Processing CreateTablet for tablet ee18915c8f6c4805a71450a47bae1a79 (DEFAULT_TABLE table=default.Table_2d8c6d6f376d4fe6909153c203750ac3 [id=67a5918ab64644c0ac0b6cfc5222eb03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.085152 14582 tablet_service.cc:1505] Processing CreateTablet for tablet ee18915c8f6c4805a71450a47bae1a79 (DEFAULT_TABLE table=default.Table_2d8c6d6f376d4fe6909153c203750ac3 [id=67a5918ab64644c0ac0b6cfc5222eb03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.085970 14427 tablet_service.cc:1505] Processing CreateTablet for tablet ee18915c8f6c4805a71450a47bae1a79 (DEFAULT_TABLE table=default.Table_2d8c6d6f376d4fe6909153c203750ac3 [id=67a5918ab64644c0ac0b6cfc5222eb03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.086019 14262 tablet_service.cc:1505] Processing CreateTablet for tablet 8ce8104b04ca4cdd8e2a9c06db9870c7 (DEFAULT_TABLE table=default.Table_f1c4e660ec174fa5b59704d5c6d8483c [id=d0b44c9f837e44cbb2e9b642ba14889d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.100450 14722 tablet_bootstrap.cc:492] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:51.100977 14722 ts_tablet_manager.cc:1403] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.457s user 0.009s sys 0.000s
I20251211 08:57:51.104735 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee18915c8f6c4805a71450a47bae1a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.115558 14714 consensus_queue.cc:260] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.116297 15650 raft_consensus.cc:493] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:51.116675 15650 raft_consensus.cc:515] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.120213 15650 leader_election.cc:290] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:51.122898 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88860fa9d7e6414185f0a1cbc78b2571" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:51.123982 14382 leader_election.cc:343] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:51.128857 14714 ts_tablet_manager.cc:1434] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.075s user 0.005s sys 0.000s
I20251211 08:57:51.130208 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4849a22585f24e388e3d8d6f4277c997. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.133069 14261 tablet_service.cc:1505] Processing CreateTablet for tablet 5aa28a4ff8c1445f873700842ad32d73 (DEFAULT_TABLE table=default.Table_7ad765fbc3eb42568ff4edb8ae8223a5 [id=2d7f0fb2f5b84e77b9cd1d5fd73d3b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.135281 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee18915c8f6c4805a71450a47bae1a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.135413 14260 tablet_service.cc:1505] Processing CreateTablet for tablet b3ad595efe5241e8a4f40b16be2ccd63 (DEFAULT_TABLE table=default.Table_fee4bd06cd7e4df08fa4bafdd291a3d4 [id=e754a76685124291a154d2a35952abd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.103122 14722 raft_consensus.cc:359] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.139271 14722 raft_consensus.cc:385] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:51.139477 14722 raft_consensus.cc:740] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:51.140946 14577 tablet_service.cc:1505] Processing CreateTablet for tablet 5aa28a4ff8c1445f873700842ad32d73 (DEFAULT_TABLE table=default.Table_7ad765fbc3eb42568ff4edb8ae8223a5 [id=2d7f0fb2f5b84e77b9cd1d5fd73d3b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.143679 14722 consensus_queue.cc:260] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.155067 14722 ts_tablet_manager.cc:1434] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20251211 08:57:51.157543 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aa28a4ff8c1445f873700842ad32d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.157541 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88860fa9d7e6414185f0a1cbc78b2571" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:51.159250 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 8ce8104b04ca4cdd8e2a9c06db9870c7 (DEFAULT_TABLE table=default.Table_f1c4e660ec174fa5b59704d5c6d8483c [id=d0b44c9f837e44cbb2e9b642ba14889d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.160270 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce8104b04ca4cdd8e2a9c06db9870c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.163384 14262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce8104b04ca4cdd8e2a9c06db9870c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.175830 15699 consensus_queue.cc:1048] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:51.177112 14261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aa28a4ff8c1445f873700842ad32d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.177738 14904 ts_tablet_manager.cc:1929] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:51.178064 14904 log.cc:1199] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/3c2fb2d9564c43ee84b8dfcea8ec54b0
I20251211 08:57:51.179520 15030 consensus_queue.cc:260] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.182111 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ad595efe5241e8a4f40b16be2ccd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.188833 14263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee18915c8f6c4805a71450a47bae1a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.192289 14273 tablet_service.cc:1505] Processing CreateTablet for tablet d3dcff38e49f46aa87ad8a30a84cc30e (DEFAULT_TABLE table=default.Table_bbb3bbff67a045babf963ea77831cda0 [id=bf6abe50458e4b39a863f26246dd3d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.193416 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3dcff38e49f46aa87ad8a30a84cc30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.183158 15030 ts_tablet_manager.cc:1434] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.186s user 0.005s sys 0.002s
I20251211 08:57:51.202730 14272 tablet_service.cc:1505] Processing CreateTablet for tablet 7831cd0a563a4f239b7f2d1c8c333b61 (DEFAULT_TABLE table=default.Table_b3c68d73c7154f10856f5fe4e01e3e8a [id=791f1a88bc3140a4b8822b10ffedf865]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.203727 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7831cd0a563a4f239b7f2d1c8c333b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.207989 14278 tablet_service.cc:1505] Processing CreateTablet for tablet ddbc5b3fad2e43b481ff733b5bad189a (DEFAULT_TABLE table=default.Table_e3591ad0b25d41499ce64dc2ea8ff7b9 [id=976c44c6409d4de69ffb74045af19d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.208918 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddbc5b3fad2e43b481ff733b5bad189a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.211408 14586 tablet_service.cc:1505] Processing CreateTablet for tablet d3dcff38e49f46aa87ad8a30a84cc30e (DEFAULT_TABLE table=default.Table_bbb3bbff67a045babf963ea77831cda0 [id=bf6abe50458e4b39a863f26246dd3d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.212486 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3dcff38e49f46aa87ad8a30a84cc30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.214787 14577 tablet_service.cc:1505] Processing CreateTablet for tablet ddbc5b3fad2e43b481ff733b5bad189a (DEFAULT_TABLE table=default.Table_e3591ad0b25d41499ce64dc2ea8ff7b9 [id=976c44c6409d4de69ffb74045af19d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.215476 14722 tablet_bootstrap.cc:492] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:51.215691 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddbc5b3fad2e43b481ff733b5bad189a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.223366 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 7831cd0a563a4f239b7f2d1c8c333b61 (DEFAULT_TABLE table=default.Table_b3c68d73c7154f10856f5fe4e01e3e8a [id=791f1a88bc3140a4b8822b10ffedf865]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.224313 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7831cd0a563a4f239b7f2d1c8c333b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.190724 14575 tablet_service.cc:1505] Processing CreateTablet for tablet b3ad595efe5241e8a4f40b16be2ccd63 (DEFAULT_TABLE table=default.Table_fee4bd06cd7e4df08fa4bafdd291a3d4 [id=e754a76685124291a154d2a35952abd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.229152 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_70f34b083c7d4203859066950bca5c6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:51.230535 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ad595efe5241e8a4f40b16be2ccd63. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:51.230556 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70f34b083c7d4203859066950bca5c6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:51.233881 14271 tablet_service.cc:1505] Processing CreateTablet for tablet 361efe27fde244289c55f203e6c420e8 (DEFAULT_TABLE table=default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.231557 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e4cd2059809846e78020a7513c96df7e" } new_table_name: "default.Table_a4f1e098ef3e4dc18569e8c2eaac9733" modify_external_catalogs: true
I20251211 08:57:51.235170 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361efe27fde244289c55f203e6c420e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.134238 14416 tablet_service.cc:1505] Processing CreateTablet for tablet 5aa28a4ff8c1445f873700842ad32d73 (DEFAULT_TABLE table=default.Table_7ad765fbc3eb42568ff4edb8ae8223a5 [id=2d7f0fb2f5b84e77b9cd1d5fd73d3b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.136713 14423 tablet_service.cc:1505] Processing CreateTablet for tablet b3ad595efe5241e8a4f40b16be2ccd63 (DEFAULT_TABLE table=default.Table_fee4bd06cd7e4df08fa4bafdd291a3d4 [id=e754a76685124291a154d2a35952abd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.238718 14416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aa28a4ff8c1445f873700842ad32d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.243206 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ad595efe5241e8a4f40b16be2ccd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.178632 14904 ts_tablet_manager.cc:1950] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:51.244869 15030 tablet_bootstrap.cc:492] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:51.251066 14714 tablet_bootstrap.cc:492] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:51.141654 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 8ce8104b04ca4cdd8e2a9c06db9870c7 (DEFAULT_TABLE table=default.Table_f1c4e660ec174fa5b59704d5c6d8483c [id=d0b44c9f837e44cbb2e9b642ba14889d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:51.158725 14385 leader_election.cc:343] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:51.264902 14385 leader_election.cc:304] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:51.265458 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce8104b04ca4cdd8e2a9c06db9870c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:51.265677 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 107042us
I20251211 08:57:51.265901 15696 raft_consensus.cc:2749] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:51.269155 15755 raft_consensus.cc:493] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:51.269660 15755 raft_consensus.cc:515] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.270586 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 361efe27fde244289c55f203e6c420e8 (DEFAULT_TABLE table=default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.271605 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361efe27fde244289c55f203e6c420e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.222975 15704 raft_consensus.cc:493] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:51.272256 15704 raft_consensus.cc:515] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.273994 15704 leader_election.cc:290] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:51.276067 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07aef1eea7c04d9cbf35c82e45df7a16" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:51.276552 14608 raft_consensus.cc:2468] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:51.277691 14382 leader_election.cc:304] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:51.278363 15696 raft_consensus.cc:2804] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:51.278681 15696 raft_consensus.cc:493] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:51.279178 15696 raft_consensus.cc:3060] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:51.285852 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 7831cd0a563a4f239b7f2d1c8c333b61 (DEFAULT_TABLE table=default.Table_b3c68d73c7154f10856f5fe4e01e3e8a [id=791f1a88bc3140a4b8822b10ffedf865]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.286186 14417 tablet_service.cc:1505] Processing CreateTablet for tablet ddbc5b3fad2e43b481ff733b5bad189a (DEFAULT_TABLE table=default.Table_e3591ad0b25d41499ce64dc2ea8ff7b9 [id=976c44c6409d4de69ffb74045af19d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.286823 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7831cd0a563a4f239b7f2d1c8c333b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.287616 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 361efe27fde244289c55f203e6c420e8 (DEFAULT_TABLE table=default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.289177 14417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddbc5b3fad2e43b481ff733b5bad189a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.293958 14905 tablet_replica.cc:333] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:51.295269 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:51.271476 15755 leader_election.cc:290] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:51.298683 14542 leader_election.cc:343] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:51.312115 14009 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
I20251211 08:57:51.313063 14009 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 4b9ae463731d4828afc02039e0d161b5 (table default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]). Expected version 1 got 0
W20251211 08:57:51.315603 14013 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 90) took 759 ms (client timeout 1000 ms). Trace:
I20251211 08:57:51.318723 14007 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
I20251211 08:57:51.334434 14905 raft_consensus.cc:2243] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:51.335492 14905 raft_consensus.cc:2272] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:51.337131 15299 ts_tablet_manager.cc:1929] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:51.339512 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:51.343341 15299 log.cc:1199] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/e15483f539f24f1ba1b442663d66fd45
I20251211 08:57:51.344818 15299 ts_tablet_manager.cc:1950] T e15483f539f24f1ba1b442663d66fd45 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
W20251211 08:57:51.316072 14013 rpcz_store.cc:269] 1211 08:57:50.556424 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:50.556528 (+ 104us) service_pool.cc:225] Handling call
1211 08:57:51.315550 (+759022us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:51.020988 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:51.021521 (+ 533us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:51.021553 (+ 32us) write_op.cc:435] Acquired schema lock
1211 08:57:51.021571 (+ 18us) tablet.cc:660] Decoding operations
1211 08:57:51.021960 (+ 389us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:51.021995 (+ 35us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:51.022015 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:51.022321 (+ 306us) tablet.cc:699] Row locks acquired
1211 08:57:51.022340 (+ 19us) write_op.cc:260] PREPARE: finished
1211 08:57:51.022463 (+ 123us) write_op.cc:270] Start()
1211 08:57:51.022559 (+ 96us) write_op.cc:276] Timestamp: P: 1765443471022554 usec, L: 0
1211 08:57:51.022571 (+ 12us) op_driver.cc:348] REPLICATION: starting
1211 08:57:51.025897 (+ 3326us) log.cc:844] Serialized 907 byte log entry
1211 08:57:51.298424 (+272527us) op_driver.cc:464] REPLICATION: finished
1211 08:57:51.299002 (+ 578us) write_op.cc:301] APPLY: starting
1211 08:57:51.299053 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:51.299234 (+ 181us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:51.299313 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:51.299576 (+ 263us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:51.299722 (+ 146us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:51.299761 (+ 39us) write_op.cc:312] APPLY: finished
1211 08:57:51.299915 (+ 154us) log.cc:844] Serialized 29 byte log entry
1211 08:57:51.300076 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:51.300143 (+ 67us) write_op.cc:454] Released schema lock
1211 08:57:51.300172 (+ 29us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:51.300418 (+ 246us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:51.300453 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":239,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":425,"apply.run_cpu_time_us":1291,"apply.run_wall_time_us":1360,"num_ops":2,"prepare.queue_time_us":263757,"prepare.run_cpu_time_us":2256,"prepare.run_wall_time_us":6023,"replication_time_us":275725}]]}
I20251211 08:57:51.368223 14905 ts_tablet_manager.cc:1916] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:51.389616 14722 tablet_bootstrap.cc:654] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:51.393649 14544 leader_election.cc:343] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:51.394042 14544 leader_election.cc:304] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:51.395090 15755 raft_consensus.cc:2749] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:51.410993 14269 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7775994f684326b6312c802f6b084d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:51.421985 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07aef1eea7c04d9cbf35c82e45df7a16" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:51.425671 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48575fa3f45349b9887bfc1cdfebc958 consensus state has no leader
I20251211 08:57:51.426713 14008 master_service.cc:954] ReplaceTablet: received request to replace tablet da124c4afb5a402fb212d0ea95e0b47e from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:51.428857 14010 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 63492ba35e26450f9645379a6501d544
I20251211 08:57:51.430344 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 29869ac852e64489973feb8b0d86ab57
I20251211 08:57:51.442780 14010 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55614 (request call id 503) took 1265 ms. Trace:
I20251211 08:57:51.447031 14006 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55614 (request call id 494) took 1703 ms. Trace:
I20251211 08:57:51.452739 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 63492ba35e26450f9645379a6501d544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:50 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:51.453675 14385 leader_election.cc:343] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:51.460506 14576 tablet_service.cc:1552] Processing DeleteTablet for tablet 63492ba35e26450f9645379a6501d544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:50 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:51.461133 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 29869ac852e64489973feb8b0d86ab57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:51.462451 14269 tablet_service.cc:1552] Processing DeleteTablet for tablet 29869ac852e64489973feb8b0d86ab57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:51.443158 14010 rpcz_store.cc:276] 1211 08:57:50.177003 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:50.528083 (+351080us) service_pool.cc:225] Handling call
1211 08:57:50.529431 (+ 1348us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:50.530576 (+ 1145us) catalog_manager.cc:2815] Modifying in-memory table state
1211 08:57:50.530683 (+ 107us) catalog_manager.cc:2824] Locking tablets
1211 08:57:50.707409 (+176726us) catalog_manager.cc:2838] Removing table and tablets from system table
1211 08:57:51.297645 (+590236us) catalog_manager.cc:2856] Removing table from by-name map
1211 08:57:51.428681 (+131036us) catalog_manager.cc:2872] Committing in-memory state
1211 08:57:51.428758 (+ 77us) catalog_manager.cc:2876] Aborting table tasks
1211 08:57:51.442723 (+ 13965us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:50.877797 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:50.878153 (+ 356us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:50.878172 (+ 19us) write_op.cc:435] Acquired schema lock
1211 08:57:50.878184 (+ 12us) tablet.cc:660] Decoding operations
1211 08:57:50.878426 (+ 242us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:50.878456 (+ 30us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:50.878476 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:50.878698 (+ 222us) tablet.cc:699] Row locks acquired
1211 08:57:50.878711 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:50.890079 (+ 11368us) write_op.cc:270] Start()
1211 08:57:50.890601 (+ 522us) write_op.cc:276] Timestamp: P: 1765443470890105 usec, L: 0
1211 08:57:50.890634 (+ 33us) op_driver.cc:348] REPLICATION: starting
1211 08:57:51.014827 (+124193us) log.cc:844] Serialized 728 byte log entry
1211 08:57:51.020722 (+ 5895us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbef5a1 00007fc20bc81aa8 00007fc20c0f34d1 00007fc20c0f2ca8 00007fc20c0f7941 00007fc20c0f78f9 00007fc20c0f7889 00007fc20c0f7851 00007fc20c0f6a7d 00000000005aa2b4 00000000005aa0e9
1211 08:57:51.264259 (+243537us) op_driver.cc:464] REPLICATION: finished
1211 08:57:51.295239 (+ 30980us) write_op.cc:301] APPLY: starting
1211 08:57:51.295498 (+ 259us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:51.295662 (+ 164us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:51.295715 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:51.296211 (+ 496us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:51.296468 (+ 257us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:51.296542 (+ 74us) write_op.cc:312] APPLY: finished
1211 08:57:51.296850 (+ 308us) log.cc:844] Serialized 29 byte log entry
1211 08:57:51.297109 (+ 259us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:51.297242 (+ 133us) write_op.cc:454] Released schema lock
1211 08:57:51.297305 (+ 63us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:51.297748 (+ 443us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:51.297797 (+ 49us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":733056,"child_traces":[["op",{"apply.queue_time_us":30755,"apply.run_cpu_time_us":1717,"apply.run_wall_time_us":2391,"num_ops":2,"prepare.queue_time_us":168768,"prepare.run_cpu_time_us":3864,"prepare.run_wall_time_us":143021,"replication_time_us":373532,"spinlock_wait_cycles":284612992}]]}
I20251211 08:57:51.447335 14006 rpcz_store.cc:276] 1211 08:57:49.791437 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.798406 (+ 6969us) service_pool.cc:225] Handling call
1211 08:57:49.800067 (+ 1661us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:49.800775 (+ 708us) catalog_manager.cc:2815] Modifying in-memory table state
1211 08:57:49.800977 (+ 202us) catalog_manager.cc:2824] Locking tablets
1211 08:57:50.701654 (+900677us) catalog_manager.cc:2838] Removing table and tablets from system table
1211 08:57:51.294364 (+592710us) catalog_manager.cc:2856] Removing table from by-name map
1211 08:57:51.430199 (+135835us) catalog_manager.cc:2872] Committing in-memory state
1211 08:57:51.430278 (+ 79us) catalog_manager.cc:2876] Aborting table tasks
1211 08:57:51.446978 (+ 16700us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:50.875945 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:50.876289 (+ 344us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:50.876310 (+ 21us) write_op.cc:435] Acquired schema lock
1211 08:57:50.876321 (+ 11us) tablet.cc:660] Decoding operations
1211 08:57:50.876592 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:50.876642 (+ 50us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:50.876662 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:50.876904 (+ 242us) tablet.cc:699] Row locks acquired
1211 08:57:50.876917 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:50.877010 (+ 93us) write_op.cc:270] Start()
1211 08:57:50.877083 (+ 73us) write_op.cc:276] Timestamp: P: 1765443470877053 usec, L: 0
1211 08:57:50.877096 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:50.877525 (+ 429us) log.cc:844] Serialized 728 byte log entry
1211 08:57:51.263002 (+385477us) op_driver.cc:464] REPLICATION: finished
1211 08:57:51.283915 (+ 20913us) write_op.cc:301] APPLY: starting
1211 08:57:51.283972 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:51.284072 (+ 100us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:51.284170 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:51.284420 (+ 250us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:51.284554 (+ 134us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:51.284591 (+ 37us) write_op.cc:312] APPLY: finished
1211 08:57:51.284770 (+ 179us) log.cc:844] Serialized 29 byte log entry
1211 08:57:51.284931 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:51.284992 (+ 61us) write_op.cc:454] Released schema lock
1211 08:57:51.285020 (+ 28us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:51.294700 (+ 9680us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:51.294753 (+ 53us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1325,"apply.run_cpu_time_us":1414,"apply.run_wall_time_us":10718,"mutex_wait_us":2622,"num_ops":2,"prepare.queue_time_us":172638,"prepare.run_cpu_time_us":1765,"prepare.run_wall_time_us":1763,"replication_time_us":385798}]]}
I20251211 08:57:51.474103 14012 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 4b9ae463731d4828afc02039e0d161b5 (table default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]). Expected version 1 got 0
I20251211 08:57:51.475018 14012 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:57:51.478201 14007 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_34b8d290de5444d49bed8a5b6ec99f05 [id=26c0a9d9b1f84372b18a54f045eb768c] alter complete (version 5)
I20251211 08:57:51.479732 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet 63492ba35e26450f9645379a6501d544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:50 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:51.481552 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet 29869ac852e64489973feb8b0d86ab57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:51.484989 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:51.485780 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet da124c4afb5a402fb212d0ea95e0b47e
I20251211 08:57:51.487543 14008 catalog_manager.cc:6724] ReplaceTablet: tablet da124c4afb5a402fb212d0ea95e0b47e deleted and replaced by tablet f6e5f7e0ce4e4e9a9f98fdb1d81d27c4
I20251211 08:57:51.487848 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet da124c4afb5a402fb212d0ea95e0b47e with delete_type TABLET_DATA_DELETED (replaced by tablet f6e5f7e0ce4e4e9a9f98fdb1d81d27c4) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:51.494266 14269 tablet_service.cc:1552] Processing DeleteTablet for tablet da124c4afb5a402fb212d0ea95e0b47e with delete_type TABLET_DATA_DELETED (replaced by tablet f6e5f7e0ce4e4e9a9f98fdb1d81d27c4) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:51.500694 15704 consensus_queue.cc:1048] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:51.502825 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet 98d14ba15dcd47f8b9e8035b3c7d477c with delete_type TABLET_DATA_DELETED (replaced by tablet 8d39234b58734161b3bcfb67d0df1bd4) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:51.503809 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7775994f684326b6312c802f6b084d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:51.504894 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet 49015c5d4f2b4d07b547a9df57c992eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:51.505884 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ab48f6288494005ad42208e2e97ddf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:48 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:51.524411 15696 raft_consensus.cc:515] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.527365 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07aef1eea7c04d9cbf35c82e45df7a16" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:51.528167 14600 raft_consensus.cc:3060] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:51.537839 14600 raft_consensus.cc:2468] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
W20251211 08:57:51.540609 13999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111) [suppressed 3 similar messages]
I20251211 08:57:51.543815 15819 raft_consensus.cc:493] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:51.549782 15819 raft_consensus.cc:515] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.551476 15821 tablet.cc:1722] T 41d6ef975a47489385791065bb8f9f46 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251211 08:57:51.555523 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_3c7e457251e74eb7b8415bbcb6b6d7db" } new_table_name: "default.Table_b6a55eca0520454c94dacf9540cb4196" modify_external_catalogs: true
W20251211 08:57:51.563951 13999 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91 -> Peer 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015): Couldn't send request to peer 73702d54989f4959acfc4a6fdcede6ca. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.62:39015: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20251211 08:57:51.552925 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361efe27fde244289c55f203e6c420e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.576172 15819 leader_election.cc:290] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:51.578996 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet da124c4afb5a402fb212d0ea95e0b47e with delete_type TABLET_DATA_DELETED (replaced by tablet f6e5f7e0ce4e4e9a9f98fdb1d81d27c4) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:51.579648 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet dd14de369e084828afb8a924c46c649b with delete_type TABLET_DATA_DELETED (replaced by tablet ee50f29bfe674611a834d84a7c4e9070) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:51.562841 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 433) took 2268 ms. Trace:
I20251211 08:57:51.580152 15299 rpcz_store.cc:276] 1211 08:57:49.294190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.295064 (+ 874us) service_pool.cc:225] Handling call
1211 08:57:51.011229 (+1716165us) ts_tablet_manager.cc:1151] Deleting tablet e15483f539f24f1ba1b442663d66fd45
1211 08:57:51.011269 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:51.285520 (+274251us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:51.336889 (+ 51369us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:51.562791 (+225902us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1715630}
I20251211 08:57:51.563182 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "552de4c6d030418181f0400714d9ec31" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:51.582369 14231 leader_election.cc:343] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:51.587759 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 438) took 2035 ms. Trace:
I20251211 08:57:51.588017 15299 rpcz_store.cc:276] 1211 08:57:49.552289 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.552393 (+ 104us) service_pool.cc:225] Handling call
1211 08:57:51.581097 (+2028704us) ts_tablet_manager.cc:1151] Deleting tablet c03f976b926a4a6fba8b70e9df9a4160
1211 08:57:51.587425 (+ 6328us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:51.587702 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2028229}
I20251211 08:57:51.591025 15299 tablet_replica.cc:333] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:51.591858 15299 raft_consensus.cc:2243] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:51.593017 15299 raft_consensus.cc:2272] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:51.595216 15299 ts_tablet_manager.cc:1916] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:51.597862 14382 consensus_peers.cc:597] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98d14ba15dcd47f8b9e8035b3c7d477c. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:51.565680 14428 tablet_service.cc:1505] Processing CreateTablet for tablet d3dcff38e49f46aa87ad8a30a84cc30e (DEFAULT_TABLE table=default.Table_bbb3bbff67a045babf963ea77831cda0 [id=bf6abe50458e4b39a863f26246dd3d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:51.627266 15819 consensus_queue.cc:1048] T 4cbd18e927c843e7b0aee7e0ccb1b06d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:51.629575 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_50297110fe8f432b9202ae4060127653" } new_table_name: "default.Table_ee7acb72cac844839994b4a220343f35" modify_external_catalogs: true
I20251211 08:57:51.633533 14722 tablet_bootstrap.cc:492] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:51.633991 14722 ts_tablet_manager.cc:1403] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.419s user 0.010s sys 0.006s
I20251211 08:57:51.633960 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_331eb0e6f45f4df69d7f5752e81e9453"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:51.634812 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7d1e362fc62d433c876de8400a9f30af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:51.635426 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_331eb0e6f45f4df69d7f5752e81e9453 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:51.636355 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d1e362fc62d433c876de8400a9f30af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:51.638447 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4575df1d27154c31927f87fb0346f042"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:51.639743 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4575df1d27154c31927f87fb0346f042 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:51.650777 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07aef1eea7c04d9cbf35c82e45df7a16" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:51.653918 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_49a263a32c9940a5b61d8a430ebb61a7 [id=69dd3ef114034ac7a3d60b8a0fde70b6]) successfully deleted
W20251211 08:57:51.654533 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet c03f976b926a4a6fba8b70e9df9a4160 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c03f976b926a4a6fba8b70e9df9a4160 already in progress: creating tablet
I20251211 08:57:51.671850 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_835755f0e12346f38507b3c844df43da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:51.673327 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_835755f0e12346f38507b3c844df43da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:51.675233 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_5a729546e47b4fc3b53ef93b18b3e438" } modify_external_catalogs: true
I20251211 08:57:51.675706 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_5a729546e47b4fc3b53ef93b18b3e438" } modify_external_catalogs: true
I20251211 08:57:51.679785 15812 consensus_queue.cc:1048] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:51.681681 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_3867d5bf46e94dd1990e38dd47037918" } modify_external_catalogs: true
I20251211 08:57:51.682240 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_3867d5bf46e94dd1990e38dd47037918" } modify_external_catalogs: true
I20251211 08:57:51.649336 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3dcff38e49f46aa87ad8a30a84cc30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:51.702872 15696 leader_election.cc:290] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:51.707892 14262 consensus_meta.cc:287] T 8ce8104b04ca4cdd8e2a9c06db9870c7 P 31404d5f58e548519239569b150d975b: Time spent flushing consensus metadata: real 0.523s user 0.007s sys 0.000s
W20251211 08:57:51.709162 14260 consensus_meta.cc:287] T b3ad595efe5241e8a4f40b16be2ccd63 P 31404d5f58e548519239569b150d975b: Time spent flushing consensus metadata: real 0.504s user 0.007s sys 0.000s
I20251211 08:57:51.710717 15828 tablet.cc:1722] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20251211 08:57:51.718856 13902 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 49879.8us
W20251211 08:57:51.719784 14385 leader_election.cc:343] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:51.707449 14714 tablet_bootstrap.cc:654] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:51.712395 15812 consensus_queue.cc:1048] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:51.724524 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 384) took 3136 ms. Trace:
W20251211 08:57:51.771219 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 48575fa3f45349b9887bfc1cdfebc958 (table default.Table_a4f1e098ef3e4dc18569e8c2eaac9733 [id=1b3efd8519a04b13af86aa631f596f33], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 48575fa3f45349b9887bfc1cdfebc958 consensus state has no leader
W20251211 08:57:51.740116 14230 consensus_peers.cc:597] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f7775994f684326b6312c802f6b084d. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:51.542723 15820 tablet.cc:1722] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251211 08:57:51.562141 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "552de4c6d030418181f0400714d9ec31" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:51.780041 14608 raft_consensus.cc:2468] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:51.630606 15030 tablet_bootstrap.cc:654] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:51.794395 14382 leader_election.cc:304] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:51.795660 15650 raft_consensus.cc:2804] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:51.796896 15650 raft_consensus.cc:697] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:51.797973 15650 consensus_queue.cc:237] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.823222 14714 tablet_bootstrap.cc:492] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:51.823817 14714 ts_tablet_manager.cc:1403] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.585s user 0.009s sys 0.001s
I20251211 08:57:51.862593 14714 raft_consensus.cc:359] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.863379 14714 raft_consensus.cc:385] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:51.863689 14714 raft_consensus.cc:740] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:51.864423 14714 consensus_queue.cc:260] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.872545 15030 tablet_bootstrap.cc:492] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:51.873032 15030 ts_tablet_manager.cc:1403] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.672s user 0.010s sys 0.000s
I20251211 08:57:51.768850 14904 rpcz_store.cc:276] 1211 08:57:48.588395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.588500 (+ 105us) service_pool.cc:225] Handling call
1211 08:57:50.538605 (+1950105us) ts_tablet_manager.cc:1151] Deleting tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0
1211 08:57:50.538642 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:50.783745 (+245103us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:51.177509 (+393764us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:51.724462 (+546953us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9648,"spinlock_wait_cycles":1792,"tablet-delete.queue_time_us":1949680}
I20251211 08:57:51.781255 14230 leader_election.cc:304] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
W20251211 08:57:51.888676 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 107574us
I20251211 08:57:51.890070 14904 tablet_replica.cc:333] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:51.890827 14904 raft_consensus.cc:2243] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:51.891294 14904 raft_consensus.cc:2272] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:51.888929 14714 ts_tablet_manager.cc:1434] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.064s user 0.005s sys 0.000s
I20251211 08:57:51.892620 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 (table default.Table_d8ac34f0cab345dd9b113ef1447ea361 [id=16e3304bc6774f7dab11c7ebd6eaa771]) successfully deleted
I20251211 08:57:51.752625 14722 raft_consensus.cc:359] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.893937 14722 raft_consensus.cc:385] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:51.894227 14722 raft_consensus.cc:740] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:51.895288 14722 consensus_queue.cc:260] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:51.746753 14278 consensus_meta.cc:287] T ddbc5b3fad2e43b481ff733b5bad189a P 31404d5f58e548519239569b150d975b: Time spent flushing consensus metadata: real 0.509s user 0.006s sys 0.000s
I20251211 08:57:51.905786 15299 ts_tablet_manager.cc:1929] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251211 08:57:51.912590 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61775901e28946ddb723f4a01f90feb6 consensus state has no leader
I20251211 08:57:51.913313 14008 master_service.cc:954] ReplaceTablet: received request to replace tablet ee50f29bfe674611a834d84a7c4e9070 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:51.938520 15820 tablet.cc:1722] T 6de43c843ff845a0bd4eb98b6054dd87 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:51.946961 14904 ts_tablet_manager.cc:1916] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:51.776700 15819 consensus_queue.cc:1048] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.190s
I20251211 08:57:51.903172 15815 raft_consensus.cc:493] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:51.967593 15815 raft_consensus.cc:515] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:51.916469 15699 raft_consensus.cc:2804] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:51.969584 15699 raft_consensus.cc:493] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:51.969947 15699 raft_consensus.cc:3060] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:51.972707 14231 leader_election.cc:343] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 4ab48f6288494005ad42208e2e97ddf2
I20251211 08:57:51.972445 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ab48f6288494005ad42208e2e97ddf2" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:51.974895 14722 ts_tablet_manager.cc:1434] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.340s user 0.007s sys 0.000s
I20251211 08:57:51.975171 14608 raft_consensus.cc:2393] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:51.976059 14230 leader_election.cc:304] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:51.987692 14905 ts_tablet_manager.cc:1929] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:51.972307 15815 leader_election.cc:290] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:51.999102 15819 raft_consensus.cc:3060] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:52.046983 15299 log.cc:1199] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/45c7d841bd3643a799a8b96e82d6d675
I20251211 08:57:51.912686 14714 tablet_bootstrap.cc:492] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:52.048683 15299 ts_tablet_manager.cc:1950] T 45c7d841bd3643a799a8b96e82d6d675 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:52.050238 14722 tablet_bootstrap.cc:492] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:51.933663 15821 tablet.cc:1722] T 6de43c843ff845a0bd4eb98b6054dd87 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:52.113034 14722 tablet_bootstrap.cc:654] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:52.118131 15699 raft_consensus.cc:515] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.123980 14905 log.cc:1199] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/4e813a4c73334c839bfe6af2119e259c
I20251211 08:57:52.124719 14905 ts_tablet_manager.cc:1950] T 4e813a4c73334c839bfe6af2119e259c P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:52.127017 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 343) took 7157 ms. Trace:
I20251211 08:57:52.127295 14905 rpcz_store.cc:276] 1211 08:57:44.969259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:44.969365 (+ 106us) service_pool.cc:225] Handling call
1211 08:57:51.091571 (+6122206us) ts_tablet_manager.cc:1151] Deleting tablet 4e813a4c73334c839bfe6af2119e259c
1211 08:57:51.293848 (+202277us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:51.717489 (+423641us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:51.987471 (+269982us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.126971 (+139500us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39513,"tablet-delete.queue_time_us":6121653}
I20251211 08:57:52.130107 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 4e813a4c73334c839bfe6af2119e259c (table default.Table_358b3598ddbe4cc2831ba06d3600ac09 [id=ffaa7c2a60924b0cbdc1254bfad98189]) successfully deleted
I20251211 08:57:51.988547 15030 raft_consensus.cc:359] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.131280 15030 raft_consensus.cc:385] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:52.131613 15030 raft_consensus.cc:740] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:52.132388 15030 consensus_queue.cc:260] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.135259 15030 ts_tablet_manager.cc:1434] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.262s user 0.008s sys 0.000s
I20251211 08:57:52.141149 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_07ea506691994cd3a2dc9606ed81de3f" } new_table_name: "default.Table_ea127c2426eb46ee891ec39320734259" modify_external_catalogs: true
I20251211 08:57:52.143561 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_dda8864ad51542599828719c4cb52e5b" } new_table_name: "default.Table_2ff48e3e6b214a0b8711f3bc10ed9319" modify_external_catalogs: true
I20251211 08:57:52.191708 15856 tablet.cc:1722] T 6de43c843ff845a0bd4eb98b6054dd87 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:52.186015 15819 raft_consensus.cc:2749] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:52.207206 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a197b5bcd78844abae3b6d6e25017d53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:52.208753 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a197b5bcd78844abae3b6d6e25017d53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:51.937772 14382 consensus_peers.cc:597] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:52.000962 15650 consensus_queue.cc:1048] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:52.284475 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 346937us
I20251211 08:57:52.276218 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 439) took 2704 ms. Trace:
I20251211 08:57:52.285097 14714 tablet_bootstrap.cc:700] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256: Time spent opening tablet: real 0.222s user 0.016s sys 0.037s
I20251211 08:57:52.290542 14714 tablet_bootstrap.cc:654] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:52.276224 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 205934us
I20251211 08:57:52.289295 15299 rpcz_store.cc:276] 1211 08:57:49.571509 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.572023 (+ 514us) service_pool.cc:225] Handling call
1211 08:57:51.588714 (+2016691us) ts_tablet_manager.cc:1151] Deleting tablet 45c7d841bd3643a799a8b96e82d6d675
1211 08:57:51.590838 (+ 2124us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:51.736169 (+145331us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:51.905572 (+169403us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.276153 (+370581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969307}
I20251211 08:57:52.294317 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 441) took 2254 ms. Trace:
I20251211 08:57:52.297494 15822 raft_consensus.cc:493] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:52.297923 15822 raft_consensus.cc:515] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.299715 15822 leader_election.cc:290] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:52.301919 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080d1700df4a4df09acda65c6d021d96" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:52.303059 15299 rpcz_store.cc:276] 1211 08:57:50.040152 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:50.041898 (+ 1746us) service_pool.cc:225] Handling call
1211 08:57:52.293993 (+2252095us) ts_tablet_manager.cc:1151] Deleting tablet e62ae5b540aa4680b4d18fee29c39e64
1211 08:57:52.294032 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:52.294270 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2250833}
I20251211 08:57:52.303974 15299 tablet_replica.cc:333] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:52.304175 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 45c7d841bd3643a799a8b96e82d6d675 (table default.Table_f1453eb82ce2412c8208000d8ba7f240 [id=f08b1a30da03451ba864cb8f8e6d199b]) successfully deleted
I20251211 08:57:52.304631 15299 raft_consensus.cc:2243] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:52.305105 15299 raft_consensus.cc:2272] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:52.305053 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet e62ae5b540aa4680b4d18fee29c39e64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e62ae5b540aa4680b4d18fee29c39e64 already in progress: creating tablet
I20251211 08:57:52.302640 14608 raft_consensus.cc:2468] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:52.307186 14230 leader_election.cc:304] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:52.315110 15299 ts_tablet_manager.cc:1916] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:52.321727 15699 leader_election.cc:290] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:52.324796 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "552de4c6d030418181f0400714d9ec31" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:52.325273 14606 raft_consensus.cc:3060] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:52.335254 14606 raft_consensus.cc:2468] T 552de4c6d030418181f0400714d9ec31 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:52.339582 14230 leader_election.cc:304] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:52.346881 15699 raft_consensus.cc:2804] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:52.347317 15699 raft_consensus.cc:697] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:52.348034 15699 consensus_queue.cc:237] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.360284 14714 tablet_bootstrap.cc:492] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:52.360682 14714 ts_tablet_manager.cc:1403] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.453s user 0.026s sys 0.045s
I20251211 08:57:52.310073 15822 raft_consensus.cc:2804] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:52.362613 14714 raft_consensus.cc:359] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.363189 14714 raft_consensus.cc:385] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:52.363394 14714 raft_consensus.cc:740] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:52.365242 14714 consensus_queue.cc:260] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.370013 15299 ts_tablet_manager.cc:1929] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:52.370352 15299 log.cc:1199] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/ccce5a446ac9460c9c9b7de72d875d95
I20251211 08:57:52.371014 15299 ts_tablet_manager.cc:1950] T ccce5a446ac9460c9c9b7de72d875d95 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:52.376408 14714 ts_tablet_manager.cc:1434] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20251211 08:57:52.377187 14714 tablet_bootstrap.cc:492] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:52.385416 15650 raft_consensus.cc:493] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:52.385818 15650 raft_consensus.cc:515] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.387526 15650 leader_election.cc:290] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:52.362850 15822 raft_consensus.cc:493] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:52.390962 15822 raft_consensus.cc:3060] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:52.393312 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb2ff90b54544420b1aad9d2ab4132ff" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:52.394019 14608 raft_consensus.cc:2468] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:52.395167 14382 leader_election.cc:304] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:52.395885 15650 raft_consensus.cc:2804] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:52.395942 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb2ff90b54544420b1aad9d2ab4132ff" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:52.396202 15650 raft_consensus.cc:493] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:52.396469 15650 raft_consensus.cc:3060] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:52.398262 14385 leader_election.cc:343] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:52.403651 15650 raft_consensus.cc:515] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.406881 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 447) took 1670 ms. Trace:
I20251211 08:57:52.407390 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb2ff90b54544420b1aad9d2ab4132ff" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:52.407505 15299 rpcz_store.cc:276] 1211 08:57:50.736032 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:50.736201 (+ 169us) service_pool.cc:225] Handling call
1211 08:57:52.303841 (+1567640us) ts_tablet_manager.cc:1151] Deleting tablet ccce5a446ac9460c9c9b7de72d875d95
1211 08:57:52.303880 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:52.343226 (+ 39346us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.369821 (+ 26595us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.406830 (+ 37009us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23573,"tablet-delete.queue_time_us":1566998}
I20251211 08:57:52.407222 15650 leader_election.cc:290] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:52.408757 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet ccce5a446ac9460c9c9b7de72d875d95 (table default.Table_afa03065224d44a3a321c94b59f12587 [id=6d0af44f2d4542168eb2af3ead7fe1c4]) successfully deleted
I20251211 08:57:52.409525 14608 raft_consensus.cc:3060] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:52.410732 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 63492ba35e26450f9645379a6501d544 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63492ba35e26450f9645379a6501d544 already in progress: creating tablet
W20251211 08:57:52.416644 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 29869ac852e64489973feb8b0d86ab57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29869ac852e64489973feb8b0d86ab57 already in progress: creating tablet
I20251211 08:57:52.407984 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb2ff90b54544420b1aad9d2ab4132ff" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:52.418323 15299 tablet_replica.cc:333] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:52.419006 14714 tablet_bootstrap.cc:654] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:52.419999 14722 tablet_bootstrap.cc:492] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:52.420418 14722 ts_tablet_manager.cc:1403] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.370s user 0.016s sys 0.000s
W20251211 08:57:52.421057 14385 leader_election.cc:343] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:52.419009 15299 raft_consensus.cc:2243] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:52.422338 15299 raft_consensus.cc:2272] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:52.425199 14722 raft_consensus.cc:359] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.429000 15299 ts_tablet_manager.cc:1916] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:52.429344 14722 raft_consensus.cc:385] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:52.430078 15822 raft_consensus.cc:515] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.432739 15822 leader_election.cc:290] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:52.434150 14601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080d1700df4a4df09acda65c6d021d96" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:52.434823 14601 raft_consensus.cc:3060] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:52.437641 15821 tablet.cc:1722] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:52.445849 14601 raft_consensus.cc:2468] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:52.439234 15856 tablet.cc:1722] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:52.453092 14722 raft_consensus.cc:740] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:52.454818 14722 consensus_queue.cc:260] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.456293 14230 leader_election.cc:304] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:52.462874 15822 raft_consensus.cc:2804] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:52.463845 15822 raft_consensus.cc:697] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:52.465816 14722 ts_tablet_manager.cc:1434] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20251211 08:57:52.485404 15030 tablet_bootstrap.cc:492] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:52.485939 14714 tablet_bootstrap.cc:492] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:52.486326 14714 ts_tablet_manager.cc:1403] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.001s
I20251211 08:57:52.488130 14722 tablet_bootstrap.cc:492] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:52.488525 14714 raft_consensus.cc:359] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.489122 14714 raft_consensus.cc:385] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:52.489382 14714 raft_consensus.cc:740] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:52.489977 14714 consensus_queue.cc:260] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.492472 14714 ts_tablet_manager.cc:1434] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20251211 08:57:52.493408 14714 tablet_bootstrap.cc:492] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:52.502481 14714 tablet_bootstrap.cc:654] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:52.465770 15822 consensus_queue.cc:237] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.520130 14722 tablet_bootstrap.cc:654] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:52.527192 15696 raft_consensus.cc:493] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:52.527717 15696 raft_consensus.cc:515] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.529865 15696 leader_election.cc:290] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:52.532330 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080d1700df4a4df09acda65c6d021d96" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:52.534418 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "552de4c6d030418181f0400714d9ec31" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:52.534551 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e5c4bf71b44abca66b0e94995c533e" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:52.535720 14385 leader_election.cc:343] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:52.535676 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080d1700df4a4df09acda65c6d021d96" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
W20251211 08:57:52.536794 14231 leader_election.cc:343] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:52.538329 14231 leader_election.cc:343] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:52.539487 14231 leader_election.cc:343] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:52.551789 15299 ts_tablet_manager.cc:1929] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:52.551394 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_5ceb94797f2d4cd2b46dab6921f5a8c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:52.552147 15299 log.cc:1199] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/98d14ba15dcd47f8b9e8035b3c7d477c
I20251211 08:57:52.552803 15299 ts_tablet_manager.cc:1950] T 98d14ba15dcd47f8b9e8035b3c7d477c P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
W20251211 08:57:52.553198 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ceb94797f2d4cd2b46dab6921f5a8c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:52.557102 14722 tablet_bootstrap.cc:492] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:52.557488 14722 ts_tablet_manager.cc:1403] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.003s
I20251211 08:57:52.572837 14722 raft_consensus.cc:359] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.573438 14722 raft_consensus.cc:385] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:52.573688 14722 raft_consensus.cc:740] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:52.578299 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 467) took 1075 ms. Trace:
I20251211 08:57:52.578585 15299 rpcz_store.cc:276] 1211 08:57:51.502482 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.502618 (+ 136us) service_pool.cc:225] Handling call
1211 08:57:52.418180 (+915562us) ts_tablet_manager.cc:1151] Deleting tablet 98d14ba15dcd47f8b9e8035b3c7d477c
1211 08:57:52.418214 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:52.512565 (+ 94351us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.551581 (+ 39016us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.578223 (+ 26642us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":914943}
I20251211 08:57:52.579504 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 98d14ba15dcd47f8b9e8035b3c7d477c (table default.Table_a4bc9b5633444f8587bf740793fd55a2 [id=1a961233d708456bb4cf8d3fc39dc705]) successfully deleted
I20251211 08:57:52.580516 14722 consensus_queue.cc:260] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.584877 14722 ts_tablet_manager.cc:1434] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.027s user 0.004s sys 0.002s
I20251211 08:57:52.590376 15299 tablet_replica.cc:333] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:52.591145 15299 raft_consensus.cc:2243] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:52.593268 15299 raft_consensus.cc:2272] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:52.585702 14722 tablet_bootstrap.cc:492] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:52.606926 15299 ts_tablet_manager.cc:1916] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:52.608539 14428 consensus_meta.cc:287] T d3dcff38e49f46aa87ad8a30a84cc30e P b276f77f87254a5d909c5ab940a39256: Time spent flushing consensus metadata: real 0.701s user 0.008s sys 0.000s
I20251211 08:57:52.609351 14428 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40738 (request call id 461) took 1342 ms. Trace:
I20251211 08:57:52.609652 14428 rpcz_store.cc:276] 1211 08:57:51.284348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.564633 (+280285us) service_pool.cc:225] Handling call
1211 08:57:51.616578 (+ 51945us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1211 08:57:51.648981 (+ 32403us) ts_tablet_manager.cc:682] Creating new metadata...
1211 08:57:51.906360 (+257379us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.608990 (+702630us) tablet_replica.cc:206] Creating consensus instance
1211 08:57:52.609298 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:57:52.621752 14722 tablet_bootstrap.cc:654] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:52.629349 14384 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 88765.5us
I20251211 08:57:52.649164 14722 tablet_bootstrap.cc:492] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:52.649655 14722 ts_tablet_manager.cc:1403] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.003s
I20251211 08:57:52.671779 14722 raft_consensus.cc:359] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.672437 14722 raft_consensus.cc:385] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:52.672698 14722 raft_consensus.cc:740] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:52.677197 14722 consensus_queue.cc:260] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.703238 14722 ts_tablet_manager.cc:1434] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.053s user 0.006s sys 0.001s
I20251211 08:57:52.704155 14722 tablet_bootstrap.cc:492] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:52.706298 15299 ts_tablet_manager.cc:1929] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:52.706645 15299 log.cc:1199] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/5f7775994f684326b6312c802f6b084d
I20251211 08:57:52.709681 15299 ts_tablet_manager.cc:1950] T 5f7775994f684326b6312c802f6b084d P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:52.718327 14722 tablet_bootstrap.cc:654] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:52.723845 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 468) took 1220 ms. Trace:
I20251211 08:57:52.724375 15030 tablet_bootstrap.cc:654] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:52.724233 15299 rpcz_store.cc:276] 1211 08:57:51.503579 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.503678 (+ 99us) service_pool.cc:225] Handling call
1211 08:57:52.590198 (+1086520us) ts_tablet_manager.cc:1151] Deleting tablet 5f7775994f684326b6312c802f6b084d
1211 08:57:52.590251 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:52.642094 (+ 51843us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.705969 (+ 63875us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.723780 (+ 17811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085971}
I20251211 08:57:52.725419 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 5f7775994f684326b6312c802f6b084d (table default.Table_0ff1ebe3d6b144cc94137467d38b30bb [id=e5e62d49ddb749a29892b61cd4615da7]) successfully deleted
I20251211 08:57:52.727718 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 469) took 1223 ms. Trace:
I20251211 08:57:52.728072 15299 rpcz_store.cc:276] 1211 08:57:51.504585 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.504757 (+ 172us) service_pool.cc:225] Handling call
1211 08:57:52.727485 (+1222728us) ts_tablet_manager.cc:1151] Deleting tablet 49015c5d4f2b4d07b547a9df57c992eb
1211 08:57:52.727525 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:52.727671 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222187}
W20251211 08:57:52.729117 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 49015c5d4f2b4d07b547a9df57c992eb because the tablet was not found. No further retry: Not found: Tablet not found: 49015c5d4f2b4d07b547a9df57c992eb
I20251211 08:57:52.730894 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 470) took 1225 ms. Trace:
I20251211 08:57:52.731199 15299 rpcz_store.cc:276] 1211 08:57:51.505660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.505755 (+ 95us) service_pool.cc:225] Handling call
1211 08:57:52.730643 (+1224888us) ts_tablet_manager.cc:1151] Deleting tablet 4ab48f6288494005ad42208e2e97ddf2
1211 08:57:52.730683 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:52.730846 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224291}
W20251211 08:57:52.735955 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 4ab48f6288494005ad42208e2e97ddf2 because the tablet was not found. No further retry: Not found: Tablet not found: 4ab48f6288494005ad42208e2e97ddf2
I20251211 08:57:52.738327 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 472) took 1166 ms. Trace:
I20251211 08:57:52.738588 15299 rpcz_store.cc:276] 1211 08:57:51.571942 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.578878 (+ 6936us) service_pool.cc:225] Handling call
1211 08:57:52.738015 (+1159137us) ts_tablet_manager.cc:1151] Deleting tablet da124c4afb5a402fb212d0ea95e0b47e
1211 08:57:52.738049 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:52.738281 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1158621}
I20251211 08:57:52.741160 15299 tablet_replica.cc:333] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:52.741922 15299 raft_consensus.cc:2243] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:52.742327 15299 raft_consensus.cc:2272] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:52.744175 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet da124c4afb5a402fb212d0ea95e0b47e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da124c4afb5a402fb212d0ea95e0b47e already in progress: creating tablet
I20251211 08:57:52.745124 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet ee50f29bfe674611a834d84a7c4e9070
I20251211 08:57:52.746404 14008 catalog_manager.cc:6724] ReplaceTablet: tablet ee50f29bfe674611a834d84a7c4e9070 deleted and replaced by tablet c8e25dad5872400a983ff71c2c0573cf
I20251211 08:57:52.746650 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet ee50f29bfe674611a834d84a7c4e9070 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e25dad5872400a983ff71c2c0573cf) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:52.746716 14008 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55614 (request call id 519) took 1063 ms. Trace:
I20251211 08:57:52.747025 14008 rpcz_store.cc:276] 1211 08:57:51.683718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.913209 (+229491us) service_pool.cc:225] Handling call
1211 08:57:51.920639 (+ 7430us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da894f1 00007fc20b6cd227 00007fc20b6cd1b2 00007fc20b6cd0e1 00007fc20b6cd05c 00007fc20b6cc312 00007fc20998455c 00007fc209983996
1211 08:57:52.746649 (+826010us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:52.639118 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:52.639497 (+ 379us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:52.639518 (+ 21us) write_op.cc:435] Acquired schema lock
1211 08:57:52.639528 (+ 10us) tablet.cc:660] Decoding operations
1211 08:57:52.639790 (+ 262us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:52.639848 (+ 58us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:52.639877 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:52.640072 (+ 195us) tablet.cc:699] Row locks acquired
1211 08:57:52.640085 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:52.640175 (+ 90us) write_op.cc:270] Start()
1211 08:57:52.640244 (+ 69us) write_op.cc:276] Timestamp: P: 1765443472640319 usec, L: 0
1211 08:57:52.640257 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:52.640701 (+ 444us) log.cc:844] Serialized 625 byte log entry
1211 08:57:52.737573 (+ 96872us) op_driver.cc:464] REPLICATION: finished
1211 08:57:52.738002 (+ 429us) write_op.cc:301] APPLY: starting
1211 08:57:52.738054 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:52.738170 (+ 116us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:52.738272 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:52.738623 (+ 351us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:52.738865 (+ 242us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:52.738907 (+ 42us) write_op.cc:312] APPLY: finished
1211 08:57:52.739071 (+ 164us) log.cc:844] Serialized 29 byte log entry
1211 08:57:52.739221 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:52.739286 (+ 65us) write_op.cc:454] Released schema lock
1211 08:57:52.739315 (+ 29us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:52.746056 (+ 6741us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:52.746081 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4151,"spinlock_wait_cycles":14883200,"child_traces":[["op",{"apply.queue_time_us":268,"apply.run_cpu_time_us":1531,"apply.run_wall_time_us":1528,"num_ops":2,"prepare.queue_time_us":714420,"prepare.run_cpu_time_us":2021,"prepare.run_wall_time_us":106935,"raft.queue_time_us":128,"raft.run_cpu_time_us":737,"raft.run_wall_time_us":746,"replication_time_us":97223}]]}
I20251211 08:57:52.753381 15299 ts_tablet_manager.cc:1916] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:52.757941 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e90daf0f780a467e9c900b91c1615cb2 consensus state has no leader
I20251211 08:57:52.754526 14428 tablet_service.cc:1552] Processing DeleteTablet for tablet ee50f29bfe674611a834d84a7c4e9070 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e25dad5872400a983ff71c2c0573cf) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:52.772745 13902 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 90402.1us
I20251211 08:57:52.773054 14722 tablet_bootstrap.cc:492] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:52.773490 14722 ts_tablet_manager.cc:1403] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.070s user 0.008s sys 0.001s
I20251211 08:57:52.775719 14722 raft_consensus.cc:359] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.777055 14722 raft_consensus.cc:385] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:52.777300 14722 raft_consensus.cc:740] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:52.778191 14714 tablet_bootstrap.cc:492] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:52.778573 14714 ts_tablet_manager.cc:1403] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.285s user 0.060s sys 0.095s
I20251211 08:57:52.780838 14714 raft_consensus.cc:359] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.781406 14714 raft_consensus.cc:385] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:52.781625 14714 raft_consensus.cc:740] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:52.782238 14714 consensus_queue.cc:260] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.786242 14722 consensus_queue.cc:260] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.787309 14714 ts_tablet_manager.cc:1434] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.008s user 0.001s sys 0.003s
I20251211 08:57:52.788132 14714 tablet_bootstrap.cc:492] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:52.791240 14722 ts_tablet_manager.cc:1434] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.017s user 0.008s sys 0.001s
I20251211 08:57:52.801429 14904 ts_tablet_manager.cc:1929] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:52.803431 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_11ad74b68fec46538d22cec80c57ddd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:52.809386 15299 ts_tablet_manager.cc:1929] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:52.809720 15299 log.cc:1199] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/dd14de369e084828afb8a924c46c649b
W20251211 08:57:52.809684 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11ad74b68fec46538d22cec80c57ddd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:52.810288 15299 ts_tablet_manager.cc:1950] T dd14de369e084828afb8a924c46c649b P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:52.816690 14714 tablet_bootstrap.cc:654] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:52.819573 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet e90daf0f780a467e9c900b91c1615cb2 (table default.Table_ea127c2426eb46ee891ec39320734259 [id=afa43e27480743738f123b2b0660f51e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e90daf0f780a467e9c900b91c1615cb2 consensus state has no leader
I20251211 08:57:52.824704 15299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 471) took 1253 ms. Trace:
I20251211 08:57:52.832927 14714 tablet_bootstrap.cc:492] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:52.833324 14714 ts_tablet_manager.cc:1403] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.000s
I20251211 08:57:52.825040 15299 rpcz_store.cc:276] 1211 08:57:51.571610 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.579534 (+ 7924us) service_pool.cc:225] Handling call
1211 08:57:52.741009 (+1161475us) ts_tablet_manager.cc:1151] Deleting tablet dd14de369e084828afb8a924c46c649b
1211 08:57:52.741060 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:52.780295 (+ 39235us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.809184 (+ 28889us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.824649 (+ 15465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160989}
I20251211 08:57:52.834417 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet dd14de369e084828afb8a924c46c649b (table default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]) successfully deleted
W20251211 08:57:52.835788 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet ee50f29bfe674611a834d84a7c4e9070 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee50f29bfe674611a834d84a7c4e9070 already in progress: creating tablet
I20251211 08:57:52.837265 14714 raft_consensus.cc:359] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.838009 14714 raft_consensus.cc:385] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251211 08:57:52.837994 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9cc3373962574253868ababbbe78e57d consensus state has no leader
I20251211 08:57:52.838317 14714 raft_consensus.cc:740] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:52.839047 14714 consensus_queue.cc:260] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.847466 14714 ts_tablet_manager.cc:1434] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20251211 08:57:52.848385 14714 tablet_bootstrap.cc:492] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:52.853941 14714 tablet_bootstrap.cc:654] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:52.860474 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_add1d53c6e304922b7ea478f77bab064"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:52.862661 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_add1d53c6e304922b7ea478f77bab064 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:52.881865 14006 master_service.cc:954] ReplaceTablet: received request to replace tablet ca5eebf2ce3848c79ee2fc21ad2b1486 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:52.895597 15899 raft_consensus.cc:493] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:52.896567 15899 raft_consensus.cc:515] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.898725 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet ee50f29bfe674611a834d84a7c4e9070 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e25dad5872400a983ff71c2c0573cf) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:52.903292 15704 raft_consensus.cc:493] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:52.903813 15704 raft_consensus.cc:515] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.905781 15704 leader_election.cc:290] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:52.911304 15810 raft_consensus.cc:493] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:57:52.912041 15810 raft_consensus.cc:515] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.913968 15810 leader_election.cc:290] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:52.919812 15810 raft_consensus.cc:493] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:52.920543 15810 raft_consensus.cc:515] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.922603 15810 leader_election.cc:290] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:52.937528 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 345) took 7756 ms. Trace:
I20251211 08:57:52.937924 14905 rpcz_store.cc:276] 1211 08:57:45.181448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.188099 (+ 6651us) service_pool.cc:225] Handling call
1211 08:57:52.566274 (+7378175us) ts_tablet_manager.cc:1151] Deleting tablet 01310095bca4479da24185576091b748
1211 08:57:52.936809 (+370535us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:52.937361 (+ 552us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7376480}
I20251211 08:57:52.939932 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 348) took 7629 ms. Trace:
I20251211 08:57:52.940286 14905 rpcz_store.cc:276] 1211 08:57:45.310488 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.310609 (+ 121us) service_pool.cc:225] Handling call
1211 08:57:52.939244 (+7628635us) ts_tablet_manager.cc:1151] Deleting tablet 3838fd220d034b3595f351b906188339
1211 08:57:52.939354 (+ 110us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:52.939860 (+ 506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7628026}
I20251211 08:57:52.941512 14905 tablet_replica.cc:333] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:57:52.942430 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 01310095bca4479da24185576091b748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01310095bca4479da24185576091b748 already in progress: creating tablet
W20251211 08:57:52.943488 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 3838fd220d034b3595f351b906188339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3838fd220d034b3595f351b906188339 already in progress: creating tablet
W20251211 08:57:52.950505 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 64178.2us
W20251211 08:57:52.954010 14608 consensus_meta.cc:287] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063: Time spent flushing consensus metadata: real 0.544s user 0.004s sys 0.010s
I20251211 08:57:52.954344 14608 raft_consensus.cc:2468] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:52.821327 14295 raft_consensus.cc:3060] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:52.822958 14231 consensus_peers.cc:597] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f7775994f684326b6312c802f6b084d. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:52.846503 14722 tablet_bootstrap.cc:492] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:52.907737 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9905227c0a844d199362a442e7d630c" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:52.958988 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 136195us
W20251211 08:57:52.960049 14385 leader_election.cc:343] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:52.967881 14722 tablet_bootstrap.cc:654] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:52.968792 14714 tablet_bootstrap.cc:492] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:52.969298 14714 ts_tablet_manager.cc:1403] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.121s user 0.021s sys 0.015s
I20251211 08:57:52.972114 14714 raft_consensus.cc:359] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.972899 14714 raft_consensus.cc:385] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:52.973196 14714 raft_consensus.cc:740] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:52.976560 14714 consensus_queue.cc:260] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.977286 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5c5b1aebde94dea81cb37a6a6998070" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:52.977802 14444 raft_consensus.cc:2468] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:52.978678 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9905227c0a844d199362a442e7d630c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:52.979233 14445 raft_consensus.cc:2468] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:52.980150 15812 raft_consensus.cc:493] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:52.980638 15812 raft_consensus.cc:515] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:52.984117 15704 consensus_queue.cc:1048] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:52.984616 14231 leader_election.cc:343] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 45c7d841bd3643a799a8b96e82d6d675
I20251211 08:57:52.985121 15812 leader_election.cc:290] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:52.986907 14714 ts_tablet_manager.cc:1434] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.017s user 0.009s sys 0.000s
I20251211 08:57:53.003471 14714 tablet_bootstrap.cc:492] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:53.009048 14722 tablet_bootstrap.cc:492] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:53.009433 14722 ts_tablet_manager.cc:1403] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.163s user 0.011s sys 0.007s
I20251211 08:57:53.011209 14904 log.cc:1199] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/98d14ba15dcd47f8b9e8035b3c7d477c
I20251211 08:57:53.017526 14904 ts_tablet_manager.cc:1950] T 98d14ba15dcd47f8b9e8035b3c7d477c P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:53.018445 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 392) took 3987 ms. Trace:
I20251211 08:57:53.018831 14904 rpcz_store.cc:276] 1211 08:57:49.031256 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.031383 (+ 127us) service_pool.cc:225] Handling call
1211 08:57:51.889929 (+2858546us) ts_tablet_manager.cc:1151] Deleting tablet 98d14ba15dcd47f8b9e8035b3c7d477c
1211 08:57:51.889969 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:52.619133 (+729164us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:52.801031 (+181898us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:53.018380 (+217349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2858039}
I20251211 08:57:53.020429 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 98d14ba15dcd47f8b9e8035b3c7d477c (table default.Table_a4bc9b5633444f8587bf740793fd55a2 [id=1a961233d708456bb4cf8d3fc39dc705]) successfully deleted
I20251211 08:57:53.033358 14714 tablet_bootstrap.cc:654] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:52.966050 15899 leader_election.cc:290] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:52.976739 14542 leader_election.cc:343] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: f938359354b545cfbac560b7d7888b9f
I20251211 08:57:53.036054 14722 raft_consensus.cc:359] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.036648 14722 raft_consensus.cc:385] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251211 08:57:53.036530 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 59848.1us
I20251211 08:57:53.036890 14722 raft_consensus.cc:740] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:53.037285 14542 leader_election.cc:304] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:53.038007 14542 leader_election.cc:304] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:53.038712 15902 raft_consensus.cc:2804] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:53.038965 15899 raft_consensus.cc:2804] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:53.040019 14722 consensus_queue.cc:260] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.040489 15899 raft_consensus.cc:493] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:53.040849 15899 raft_consensus.cc:3060] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:53.042200 14722 ts_tablet_manager.cc:1434] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20251211 08:57:53.046911 14722 tablet_bootstrap.cc:492] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:53.051260 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5c5b1aebde94dea81cb37a6a6998070" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:53.053239 14382 leader_election.cc:304] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:53.054328 15696 raft_consensus.cc:2804] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:53.054593 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9905227c0a844d199362a442e7d630c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:53.054746 15696 raft_consensus.cc:697] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:53.055481 15696 consensus_queue.cc:237] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.068533 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45c7d841bd3643a799a8b96e82d6d675" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:53.072490 14601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e5c4bf71b44abca66b0e94995c533e" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:53.073196 14722 tablet_bootstrap.cc:654] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:53.078743 14714 tablet_bootstrap.cc:492] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:53.079262 14714 ts_tablet_manager.cc:1403] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.005s
I20251211 08:57:53.081698 14714 raft_consensus.cc:359] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.082311 14714 raft_consensus.cc:385] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:53.082569 14714 raft_consensus.cc:740] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:53.083187 14714 consensus_queue.cc:260] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.083716 14601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9905227c0a844d199362a442e7d630c" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:53.091666 14714 ts_tablet_manager.cc:1434] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.012s user 0.003s sys 0.002s
I20251211 08:57:53.092679 14714 tablet_bootstrap.cc:492] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:53.100071 14230 leader_election.cc:304] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:53.101130 15822 raft_consensus.cc:2749] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:53.101719 14382 leader_election.cc:343] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:53.102084 14382 leader_election.cc:304] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:53.102710 15650 raft_consensus.cc:2749] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:53.108284 15030 tablet_bootstrap.cc:492] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:53.108732 15696 consensus_queue.cc:1048] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:53.108873 15030 ts_tablet_manager.cc:1403] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.624s user 0.013s sys 0.000s
W20251211 08:57:53.113680 14544 leader_election.cc:343] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:53.114722 14544 leader_election.cc:343] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:53.117877 14544 leader_election.cc:343] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Not found: Tablet not found: f938359354b545cfbac560b7d7888b9f
I20251211 08:57:53.118258 14544 leader_election.cc:304] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:53.121802 15901 raft_consensus.cc:2749] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:53.114722 14714 tablet_bootstrap.cc:654] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:53.145499 15902 raft_consensus.cc:493] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:53.145891 15902 raft_consensus.cc:3060] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:53.146423 14231 consensus_peers.cc:597] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccce5a446ac9460c9c9b7de72d875d95. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:53.170992 15820 tablet.cc:1722] T 18f32a1ccedd4f15bf8c79bc3f16e6bb P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:53.183673 14714 tablet_bootstrap.cc:492] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:53.184228 14714 ts_tablet_manager.cc:1403] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.092s user 0.009s sys 0.002s
I20251211 08:57:53.187641 14714 raft_consensus.cc:359] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.188416 14714 raft_consensus.cc:385] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:53.188828 14714 raft_consensus.cc:740] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:53.189440 14714 consensus_queue.cc:260] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.193560 14714 ts_tablet_manager.cc:1434] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20251211 08:57:53.199934 15899 raft_consensus.cc:515] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.201463 14601 raft_consensus.cc:2393] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate c5e624bcfd154cd586ef1dedb9940063 in this term.
I20251211 08:57:53.205634 14382 leader_election.cc:304] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:53.206303 15696 raft_consensus.cc:3060] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:53.212255 15696 raft_consensus.cc:2749] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:53.221123 15030 raft_consensus.cc:359] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.222124 15030 raft_consensus.cc:385] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:53.222487 15030 raft_consensus.cc:740] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:53.223584 15030 consensus_queue.cc:260] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.224386 14722 tablet_bootstrap.cc:492] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:53.224828 14722 ts_tablet_manager.cc:1403] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.182s user 0.013s sys 0.010s
I20251211 08:57:53.225327 14904 tablet_replica.cc:333] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:53.226419 14904 raft_consensus.cc:2243] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:53.227048 14904 raft_consensus.cc:2272] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:53.229756 14904 ts_tablet_manager.cc:1916] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:53.230912 14722 raft_consensus.cc:359] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.231550 14722 raft_consensus.cc:385] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:53.231771 14722 raft_consensus.cc:740] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:53.232345 14722 consensus_queue.cc:260] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.240435 14714 tablet_bootstrap.cc:492] T 552de4c6d030418181f0400714d9ec31 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:53.242100 14722 ts_tablet_manager.cc:1434] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20251211 08:57:53.245493 14714 tablet_bootstrap.cc:654] T 552de4c6d030418181f0400714d9ec31 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:53.254875 14722 tablet_bootstrap.cc:492] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:53.267709 14714 tablet_bootstrap.cc:492] T 552de4c6d030418181f0400714d9ec31 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:53.268106 14714 ts_tablet_manager.cc:1403] T 552de4c6d030418181f0400714d9ec31 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.004s
I20251211 08:57:53.270190 14714 raft_consensus.cc:359] T 552de4c6d030418181f0400714d9ec31 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.270182 15812 consensus_queue.cc:1048] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:53.270746 14714 raft_consensus.cc:385] T 552de4c6d030418181f0400714d9ec31 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:53.271008 14714 raft_consensus.cc:740] T 552de4c6d030418181f0400714d9ec31 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:53.271732 14714 consensus_queue.cc:260] T 552de4c6d030418181f0400714d9ec31 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.274895 14714 ts_tablet_manager.cc:1434] T 552de4c6d030418181f0400714d9ec31 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20251211 08:57:53.275624 14714 tablet_bootstrap.cc:492] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:53.280972 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9905227c0a844d199362a442e7d630c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:53.282406 14544 leader_election.cc:343] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:53.283138 14714 tablet_bootstrap.cc:654] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:53.291234 15899 leader_election.cc:290] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:53.295372 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9905227c0a844d199362a442e7d630c" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:53.299499 14444 raft_consensus.cc:3060] T 552de4c6d030418181f0400714d9ec31 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:53.302258 15030 ts_tablet_manager.cc:1434] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.190s user 0.005s sys 0.002s
I20251211 08:57:53.305837 15030 tablet_bootstrap.cc:492] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:53.307854 15812 consensus_queue.cc:1048] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:53.316666 14447 raft_consensus.cc:2468] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:53.317860 14542 leader_election.cc:304] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:53.320513 15514 raft_consensus.cc:2804] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:53.321002 15514 raft_consensus.cc:697] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:53.321713 15514 consensus_queue.cc:237] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.323305 15812 consensus_queue.cc:1048] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251211 08:57:53.348125 14722 tablet_bootstrap.cc:654] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:53.372174 14722 tablet_bootstrap.cc:492] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:53.373819 14722 ts_tablet_manager.cc:1403] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.127s user 0.008s sys 0.001s
I20251211 08:57:53.382081 14447 raft_consensus.cc:3060] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:53.397207 14722 raft_consensus.cc:359] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.409438 14722 raft_consensus.cc:385] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:53.410194 14722 raft_consensus.cc:740] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:53.419588 14722 consensus_queue.cc:260] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.428085 14278 tablet_service.cc:1505] Processing CreateTablet for tablet 020230ff08ed42179a28efbb21613a0f (DEFAULT_TABLE table=default.Table_1df60c18431f4fcbb050908676608f64 [id=81f8537793284788a0660d02027cf529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:53.429320 14722 ts_tablet_manager.cc:1434] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.034s user 0.004s sys 0.001s
I20251211 08:57:53.429395 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 020230ff08ed42179a28efbb21613a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:53.430377 14722 tablet_bootstrap.cc:492] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:53.436369 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_bdaa13035e5f4ecb80d93694ed51149e [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 1 got 0
I20251211 08:57:53.437791 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 18f32a1ccedd4f15bf8c79bc3f16e6bb (table default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0]). Expected version 2 got 0
I20251211 08:57:53.443761 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
W20251211 08:57:53.460861 14012 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 62) took 1962 ms (client timeout 1000 ms). Trace:
I20251211 08:57:53.464041 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 020230ff08ed42179a28efbb21613a0f (DEFAULT_TABLE table=default.Table_1df60c18431f4fcbb050908676608f64 [id=81f8537793284788a0660d02027cf529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:53.465235 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 020230ff08ed42179a28efbb21613a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:53.472771 15696 raft_consensus.cc:493] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:53.473342 15696 raft_consensus.cc:515] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.475121 15696 leader_election.cc:290] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:53.478859 14013 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 93) took 1594 ms (client timeout 1000 ms). Trace:
W20251211 08:57:53.490985 14013 rpcz_store.cc:269] 1211 08:57:51.884488 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.893799 (+ 9311us) service_pool.cc:225] Handling call
1211 08:57:51.928151 (+ 34352us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:53.478758 (+1550607us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":29454208}
I20251211 08:57:53.485316 14008 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 4b9ae463731d4828afc02039e0d161b5 (table default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]). Expected version 1 got 0
I20251211 08:57:53.493997 14008 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_bdaa13035e5f4ecb80d93694ed51149e [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 1 got 0
I20251211 08:57:53.495136 14008 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_31fcfbed4a2c4e8aa05fb42c261bf1d5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 2 got 0
I20251211 08:57:53.503690 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:57:53.504973 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 4b9ae463731d4828afc02039e0d161b5 (table default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]). Expected version 1 got 0
I20251211 08:57:53.505883 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet f5c5b1aebde94dea81cb37a6a6998070 (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]). Expected version 1 got 0
W20251211 08:57:53.516005 14012 rpcz_store.cc:269] 1211 08:57:51.498485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.499589 (+ 1104us) service_pool.cc:225] Handling call
1211 08:57:53.460771 (+1961182us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":6}
I20251211 08:57:53.516752 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 4980038a4d134909ad57c2d60fee5ada (DEFAULT_TABLE table=default.Table_c57ef6202bd14b8c9ff0e4a6bbe161b2 [id=6f3923d7fe8b428fa8e3ba09bec5126a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:53.517623 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4980038a4d134909ad57c2d60fee5ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:53.523343 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_6fd146e5066b4fb9b651b986d7992aa8" } new_table_name: "default.Table_718ae82148e34936ba20f6d0bebe6169" modify_external_catalogs: true
I20251211 08:57:53.563921 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341b762afd8440bb7198ca134c8dffe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:53.564695 14600 raft_consensus.cc:2468] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:53.567462 14382 leader_election.cc:304] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:53.568363 15696 raft_consensus.cc:2804] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:53.568727 15696 raft_consensus.cc:493] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:53.569039 15696 raft_consensus.cc:3060] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:53.586831 15871 raft_consensus.cc:493] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:57:53.594233 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e813a4c73334c839bfe6af2119e259c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:53.604065 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 020230ff08ed42179a28efbb21613a0f (DEFAULT_TABLE table=default.Table_1df60c18431f4fcbb050908676608f64 [id=81f8537793284788a0660d02027cf529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:53.605451 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 020230ff08ed42179a28efbb21613a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:53.608107 15901 raft_consensus.cc:493] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:53.608588 15901 raft_consensus.cc:515] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.613130 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 785ccfbc31b74ca6969bcf4020b0520c (DEFAULT_TABLE table=default.Table_5e9e1c179cbb4953b47e0c0400ee35fc [id=e7a7c6b9f7b94aca8def832fcae1384d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:53.587402 15871 raft_consensus.cc:515] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.616714 15871 leader_election.cc:290] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:53.617175 15871 raft_consensus.cc:493] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:53.617614 15871 raft_consensus.cc:515] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.623241 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785ccfbc31b74ca6969bcf4020b0520c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:53.610661 15901 leader_election.cc:290] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:53.634989 15871 leader_election.cc:290] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:53.657406 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 4b9ae463731d4828afc02039e0d161b5 (table default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]). Expected version 1 got 0
I20251211 08:57:53.658746 14011 catalog_manager.cc:5654] T 552de4c6d030418181f0400714d9ec31 P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:53.660497 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:57:53.661916 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet da52cace2aab4f9dacef18ec5fa56aaf
I20251211 08:57:53.663352 14005 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55614 (request call id 517) took 1992 ms. Trace:
I20251211 08:57:53.667869 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet da52cace2aab4f9dacef18ec5fa56aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:51 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:53.663710 14005 rpcz_store.cc:276] 1211 08:57:51.670917 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.674868 (+ 3951us) service_pool.cc:225] Handling call
1211 08:57:51.676086 (+ 1218us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:51.677615 (+ 1529us) catalog_manager.cc:2815] Modifying in-memory table state
1211 08:57:51.677757 (+ 142us) catalog_manager.cc:2824] Locking tablets
1211 08:57:51.678544 (+ 787us) catalog_manager.cc:2838] Removing table and tablets from system table
1211 08:57:52.584028 (+905484us) catalog_manager.cc:2856] Removing table from by-name map
1211 08:57:52.584220 (+ 192us) catalog_manager.cc:2872] Committing in-memory state
1211 08:57:53.661827 (+1077607us) catalog_manager.cc:2876] Aborting table tasks
1211 08:57:53.663278 (+ 1451us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:52.557147 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:52.571317 (+ 14170us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:52.571353 (+ 36us) write_op.cc:435] Acquired schema lock
1211 08:57:52.571369 (+ 16us) tablet.cc:660] Decoding operations
1211 08:57:52.571663 (+ 294us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:52.571714 (+ 51us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:52.571738 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:52.571973 (+ 235us) tablet.cc:699] Row locks acquired
1211 08:57:52.571985 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:57:52.572115 (+ 130us) write_op.cc:270] Start()
1211 08:57:52.572190 (+ 75us) write_op.cc:276] Timestamp: P: 1765443472572227 usec, L: 0
1211 08:57:52.572203 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:52.572637 (+ 434us) log.cc:844] Serialized 728 byte log entry
1211 08:57:52.581947 (+ 9310us) op_driver.cc:464] REPLICATION: finished
1211 08:57:52.582440 (+ 493us) write_op.cc:301] APPLY: starting
1211 08:57:52.582528 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:52.582651 (+ 123us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:52.582783 (+ 132us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:52.583110 (+ 327us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:52.583259 (+ 149us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:52.583298 (+ 39us) write_op.cc:312] APPLY: finished
1211 08:57:52.583484 (+ 186us) log.cc:844] Serialized 29 byte log entry
1211 08:57:52.583618 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:52.583680 (+ 62us) write_op.cc:454] Released schema lock
1211 08:57:52.583709 (+ 29us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:52.638771 (+ 55062us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:52.638796 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":330,"apply.run_cpu_time_us":1439,"apply.run_wall_time_us":1438,"mutex_wait_us":63498,"num_ops":2,"prepare.queue_time_us":869162,"prepare.run_cpu_time_us":2152,"prepare.run_wall_time_us":81597,"raft.queue_time_us":82,"raft.run_cpu_time_us":797,"raft.run_wall_time_us":796,"replication_time_us":9655}]]}
I20251211 08:57:53.676414 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 98d14ba15dcd47f8b9e8035b3c7d477c with delete_type TABLET_DATA_DELETED (replaced by tablet 8d39234b58734161b3bcfb67d0df1bd4) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:53.678359 14010 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 56) took 2024 ms (client timeout 1000 ms). Trace:
I20251211 08:57:53.705796 14575 tablet_service.cc:1552] Processing DeleteTablet for tablet da52cace2aab4f9dacef18ec5fa56aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:51 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:53.716325 15902 consensus_meta.cc:287] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063: Time spent flushing consensus metadata: real 0.570s user 0.008s sys 0.002s
I20251211 08:57:53.716671 15902 raft_consensus.cc:515] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.736169 15650 raft_consensus.cc:493] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:53.736565 15650 raft_consensus.cc:515] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.776535 15902 leader_election.cc:290] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:53.785728 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e5c4bf71b44abca66b0e94995c533e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:53.786355 14444 raft_consensus.cc:2468] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:53.792464 15030 tablet_bootstrap.cc:654] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:53.794353 15650 leader_election.cc:290] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:53.797163 15650 raft_consensus.cc:493] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:53.798870 14905 raft_consensus.cc:2243] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:53.798750 15650 raft_consensus.cc:515] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.799373 14905 raft_consensus.cc:2272] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:53.800362 15650 leader_election.cc:290] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:53.795521 15941 tablet_replica.cc:333] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:53.801182 14905 ts_tablet_manager.cc:1916] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:53.801287 15941 raft_consensus.cc:2243] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:53.802148 15941 raft_consensus.cc:2272] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:53.804070 15941 ts_tablet_manager.cc:1916] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:53.804426 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5c5b1aebde94dea81cb37a6a6998070" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:53.811224 14444 raft_consensus.cc:3060] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:53.815685 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 45c7d841bd3643a799a8b96e82d6d675 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:53.817107 14444 raft_consensus.cc:2468] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:53.817515 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:53.818825 14714 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.047s sys 0.068s
W20251211 08:57:53.819052 14714 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.519s user 0.047s sys 0.068s
I20251211 08:57:53.818926 14417 tablet_service.cc:1505] Processing CreateTablet for tablet 785ccfbc31b74ca6969bcf4020b0520c (DEFAULT_TABLE table=default.Table_5e9e1c179cbb4953b47e0c0400ee35fc [id=e7a7c6b9f7b94aca8def832fcae1384d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:53.819988 14417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785ccfbc31b74ca6969bcf4020b0520c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:53.820771 14714 tablet_bootstrap.cc:492] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:53.820027 15650 raft_consensus.cc:493] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:53.821259 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 4980038a4d134909ad57c2d60fee5ada (DEFAULT_TABLE table=default.Table_c57ef6202bd14b8c9ff0e4a6bbe161b2 [id=6f3923d7fe8b428fa8e3ba09bec5126a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:53.821384 15650 raft_consensus.cc:515] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.822110 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4980038a4d134909ad57c2d60fee5ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:53.822888 15650 leader_election.cc:290] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:53.821128 14714 ts_tablet_manager.cc:1403] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.546s user 0.054s sys 0.068s
I20251211 08:57:53.825640 14714 raft_consensus.cc:359] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.826182 14714 raft_consensus.cc:385] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:53.826426 14714 raft_consensus.cc:740] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:53.838120 15030 tablet_bootstrap.cc:492] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:53.722002 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_1610127d3bbc43ae803baa1e8470000f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:53.838551 15030 ts_tablet_manager.cc:1403] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.533s user 0.001s sys 0.009s
W20251211 08:57:53.840008 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1610127d3bbc43ae803baa1e8470000f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:53.725495 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ac8e892bbe604ff89a5b31d2a906eaac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:53.845429 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac8e892bbe604ff89a5b31d2a906eaac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:53.842967 14714 consensus_queue.cc:260] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.726781 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_f0670566e5df45448e1786b9c82156d4" } new_table_name: "default.Table_f66d427103534bd98c3bcc5f0734b392" modify_external_catalogs: true
I20251211 08:57:53.766047 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 07099552b546444bab4d66238dd6d146
I20251211 08:57:53.848311 14004 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55614 (request call id 518) took 2167 ms. Trace:
I20251211 08:57:53.848353 15030 raft_consensus.cc:359] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.848848 15030 raft_consensus.cc:385] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:53.849042 15030 raft_consensus.cc:740] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:53.849474 15030 consensus_queue.cc:260] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.851428 15030 ts_tablet_manager.cc:1434] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20251211 08:57:53.852165 15030 tablet_bootstrap.cc:492] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:53.852438 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 07099552b546444bab4d66238dd6d146 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:51 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:53.848662 14004 rpcz_store.cc:276] 1211 08:57:51.681178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.681303 (+ 125us) service_pool.cc:225] Handling call
1211 08:57:51.682607 (+ 1304us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:51.683211 (+ 604us) catalog_manager.cc:2815] Modifying in-memory table state
1211 08:57:51.683320 (+ 109us) catalog_manager.cc:2824] Locking tablets
1211 08:57:53.416680 (+1733360us) catalog_manager.cc:2838] Removing table and tablets from system table
1211 08:57:53.765653 (+348973us) catalog_manager.cc:2856] Removing table from by-name map
1211 08:57:53.765861 (+ 208us) catalog_manager.cc:2872] Committing in-memory state
1211 08:57:53.765952 (+ 91us) catalog_manager.cc:2876] Aborting table tasks
1211 08:57:53.848247 (+ 82295us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:53.727835 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:53.728310 (+ 475us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:53.728336 (+ 26us) write_op.cc:435] Acquired schema lock
1211 08:57:53.728369 (+ 33us) tablet.cc:660] Decoding operations
1211 08:57:53.728707 (+ 338us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:53.728753 (+ 46us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:53.728780 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:53.729048 (+ 268us) tablet.cc:699] Row locks acquired
1211 08:57:53.729064 (+ 16us) write_op.cc:260] PREPARE: finished
1211 08:57:53.729173 (+ 109us) write_op.cc:270] Start()
1211 08:57:53.729257 (+ 84us) write_op.cc:276] Timestamp: P: 1765443473729325 usec, L: 0
1211 08:57:53.729275 (+ 18us) op_driver.cc:348] REPLICATION: starting
1211 08:57:53.729823 (+ 548us) log.cc:844] Serialized 728 byte log entry
1211 08:57:53.760602 (+ 30779us) op_driver.cc:464] REPLICATION: finished
1211 08:57:53.763730 (+ 3128us) write_op.cc:301] APPLY: starting
1211 08:57:53.763798 (+ 68us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:53.763919 (+ 121us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:53.764033 (+ 114us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:53.764334 (+ 301us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:53.764489 (+ 155us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:53.764541 (+ 52us) write_op.cc:312] APPLY: finished
1211 08:57:53.764733 (+ 192us) log.cc:844] Serialized 29 byte log entry
1211 08:57:53.764870 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:53.764943 (+ 73us) write_op.cc:454] Released schema lock
1211 08:57:53.764977 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2959,"apply.run_cpu_time_us":1467,"apply.run_wall_time_us":2691,"num_ops":2,"prepare.queue_time_us":309035,"raft.queue_time_us":4227,"raft.run_cpu_time_us":672,"raft.run_wall_time_us":1495,"replication_time_us":31225}]]}
I20251211 08:57:53.855261 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 07099552b546444bab4d66238dd6d146 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:51 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:53.856938 14905 ts_tablet_manager.cc:1929] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:53.857251 14905 log.cc:1199] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/3c2fb2d9564c43ee84b8dfcea8ec54b0
I20251211 08:57:53.857899 14905 ts_tablet_manager.cc:1950] T 3c2fb2d9564c43ee84b8dfcea8ec54b0 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:53.859647 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 349) took 8547 ms. Trace:
I20251211 08:57:53.859939 14905 rpcz_store.cc:276] 1211 08:57:45.311992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.312104 (+ 112us) service_pool.cc:225] Handling call
1211 08:57:52.941259 (+7629155us) ts_tablet_manager.cc:1151] Deleting tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0
1211 08:57:52.941362 (+ 103us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:53.813044 (+871682us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:53.856759 (+ 43715us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:53.859591 (+ 2832us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":856398,"tablet-delete.queue_time_us":7628690}
I20251211 08:57:53.861110 14905 tablet_replica.cc:333] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:53.861308 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 3c2fb2d9564c43ee84b8dfcea8ec54b0 (table default.Table_d8ac34f0cab345dd9b113ef1447ea361 [id=16e3304bc6774f7dab11c7ebd6eaa771]) successfully deleted
I20251211 08:57:53.861788 14905 raft_consensus.cc:2243] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
W20251211 08:57:53.863440 14010 rpcz_store.cc:269] 1211 08:57:51.653997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.654125 (+ 128us) service_pool.cc:225] Handling call
1211 08:57:53.678291 (+2024166us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8}
I20251211 08:57:53.864842 15030 tablet_bootstrap.cc:654] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:53.865861 15902 raft_consensus.cc:493] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:53.866209 15902 raft_consensus.cc:515] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.867666 15902 leader_election.cc:290] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:53.869577 14714 ts_tablet_manager.cc:1434] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.046s user 0.004s sys 0.000s
I20251211 08:57:53.870496 14714 tablet_bootstrap.cc:492] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:53.877549 14543 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 68100.2us
I20251211 08:57:53.789888 15822 raft_consensus.cc:493] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:53.892540 14602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080d1700df4a4df09acda65c6d021d96" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:53.892802 15812 raft_consensus.cc:493] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:57:53.862661 14905 raft_consensus.cc:2272] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:53.893347 15812 raft_consensus.cc:515] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.895054 15812 leader_election.cc:290] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:53.882429 15934 consensus_queue.cc:1048] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20251211 08:57:53.896296 15030 tablet_bootstrap.cc:492] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:53.887943 15822 raft_consensus.cc:515] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.896737 15030 ts_tablet_manager.cc:1403] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.003s
I20251211 08:57:53.897712 15822 leader_election.cc:290] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:53.900120 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 140033us
W20251211 08:57:53.900871 13999 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f3cf2b4d14ba45d79e52b6fdb0497d58 (table default.Table_d3f2e5059297427fbaaefd79504a2e2f [id=945a9eb17575451192d09cf17a00a46d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f3cf2b4d14ba45d79e52b6fdb0497d58 consensus state has no leader
I20251211 08:57:53.905455 15812 raft_consensus.cc:493] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:53.905886 15812 raft_consensus.cc:515] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.907549 15812 leader_election.cc:290] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:53.911023 14905 ts_tablet_manager.cc:1916] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:53.911051 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e5c4bf71b44abca66b0e94995c533e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:53.911942 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e15483f539f24f1ba1b442663d66fd45" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:53.914810 14296 raft_consensus.cc:2468] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:53.914873 15936 raft_consensus.cc:493] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:53.913259 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da57b72207414512b9744b283bc869fe" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:53.915436 15936 raft_consensus.cc:515] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.915634 15030 raft_consensus.cc:359] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.916178 15030 raft_consensus.cc:385] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:53.916491 15030 raft_consensus.cc:740] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:53.916987 15030 consensus_queue.cc:260] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.917498 14299 raft_consensus.cc:2468] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:53.919456 14261 tablet_service.cc:1505] Processing CreateTablet for tablet 785ccfbc31b74ca6969bcf4020b0520c (DEFAULT_TABLE table=default.Table_5e9e1c179cbb4953b47e0c0400ee35fc [id=e7a7c6b9f7b94aca8def832fcae1384d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:53.919373 15030 ts_tablet_manager.cc:1434] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.022s user 0.004s sys 0.002s
I20251211 08:57:53.920418 14261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785ccfbc31b74ca6969bcf4020b0520c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:53.922343 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 98d14ba15dcd47f8b9e8035b3c7d477c with delete_type TABLET_DATA_DELETED (replaced by tablet 8d39234b58734161b3bcfb67d0df1bd4) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:53.923336 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5c5b1aebde94dea81cb37a6a6998070" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:53.924566 15030 tablet_bootstrap.cc:492] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:53.924897 14271 tablet_service.cc:1505] Processing CreateTablet for tablet 4980038a4d134909ad57c2d60fee5ada (DEFAULT_TABLE table=default.Table_c57ef6202bd14b8c9ff0e4a6bbe161b2 [id=6f3923d7fe8b428fa8e3ba09bec5126a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:53.925765 14271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4980038a4d134909ad57c2d60fee5ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:53.931793 15936 leader_election.cc:290] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:53.935240 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet da52cace2aab4f9dacef18ec5fa56aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:51 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:53.937417 14260 tablet_service.cc:1552] Processing DeleteTablet for tablet 07099552b546444bab4d66238dd6d146 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:51 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:53.940094 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e05628011634149b17d298c54074d00" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:53.940621 14293 raft_consensus.cc:2468] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:53.942087 14260 tablet_service.cc:1505] Processing CreateTablet for tablet 7588198da5544af99b5fd8e25540c664 (DEFAULT_TABLE table=default.Table_466402e26ef741c8bb19f291f23f54b7 [id=a4ec3a32c3104351bbd088d559e30ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:53.942303 14905 ts_tablet_manager.cc:1929] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:53.943092 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7588198da5544af99b5fd8e25540c664. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:53.946532 14722 tablet_bootstrap.cc:654] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:53.948949 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 80582.7us
I20251211 08:57:53.949586 14542 leader_election.cc:304] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:53.950387 15902 raft_consensus.cc:2804] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:53.950775 15902 raft_consensus.cc:493] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:53.951192 15902 raft_consensus.cc:3060] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:53.952194 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341b762afd8440bb7198ca134c8dffe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:53.954643 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "080d1700df4a4df09acda65c6d021d96" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:53.956694 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:53.957651 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e05628011634149b17d298c54074d00" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:53.958053 14296 raft_consensus.cc:2468] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
W20251211 08:57:53.959180 14385 leader_election.cc:343] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:53.960008 15952 raft_consensus.cc:493] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:53.960436 15952 raft_consensus.cc:515] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.962299 15952 leader_election.cc:290] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:53.963466 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49015c5d4f2b4d07b547a9df57c992eb" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:53.963805 14544 leader_election.cc:343] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:53.964190 14605 raft_consensus.cc:2393] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:53.964375 14544 leader_election.cc:304] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:53.966404 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e15483f539f24f1ba1b442663d66fd45" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:53.966876 14606 raft_consensus.cc:2468] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:53.968302 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb2ff90b54544420b1aad9d2ab4132ff" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:53.968876 14600 raft_consensus.cc:2393] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:53.969532 14722 tablet_bootstrap.cc:492] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:53.969847 14722 ts_tablet_manager.cc:1403] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.540s user 0.007s sys 0.003s
I20251211 08:57:53.971033 15939 raft_consensus.cc:2804] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:57:53.971333 15939 raft_consensus.cc:493] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:57:53.971629 15939 raft_consensus.cc:3060] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:57:53.971815 14722 raft_consensus.cc:359] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.972321 14722 raft_consensus.cc:385] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:53.972551 14722 raft_consensus.cc:740] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:53.973250 14722 consensus_queue.cc:260] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:53.985325 14722 ts_tablet_manager.cc:1434] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20251211 08:57:53.991233 14722 tablet_bootstrap.cc:492] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:53.991255 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6c6a2b03ebbd4c7b9db039b0cfca6997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:53.992794 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c6a2b03ebbd4c7b9db039b0cfca6997 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:53.996681 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_dbdc2eaf4309442888ea49a832da3967"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:54.004539 14722 tablet_bootstrap.cc:654] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:54.014011 14231 leader_election.cc:343] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: da57b72207414512b9744b283bc869fe
W20251211 08:57:54.014631 14231 leader_election.cc:343] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 49015c5d4f2b4d07b547a9df57c992eb
W20251211 08:57:54.015254 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 55198us
W20251211 08:57:54.016793 14231 leader_election.cc:343] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: c6d0fef50524476dbd47fc70e4bda319
W20251211 08:57:54.017253 14231 leader_election.cc:343] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: e15483f539f24f1ba1b442663d66fd45
I20251211 08:57:54.020942 14722 tablet_bootstrap.cc:492] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:54.021353 14722 ts_tablet_manager.cc:1403] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20251211 08:57:54.023882 14722 raft_consensus.cc:359] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.024504 14722 raft_consensus.cc:385] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:54.024765 14722 raft_consensus.cc:740] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:54.025240 14722 consensus_queue.cc:260] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.028843 14544 leader_election.cc:304] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
W20251211 08:57:54.029661 14544 leader_election.cc:343] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:54.039680 14722 ts_tablet_manager.cc:1434] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20251211 08:57:54.036902 15514 raft_consensus.cc:2804] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:57:54.047183 15514 raft_consensus.cc:493] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:54.047580 15514 raft_consensus.cc:3060] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:57:54.049746 14722 tablet_bootstrap.cc:492] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:57:54.051199 14661 tablet.cc:2378] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251211 08:57:54.055112 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 105642us
W20251211 08:57:54.055819 14542 leader_election.cc:343] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: e15483f539f24f1ba1b442663d66fd45
I20251211 08:57:54.057480 14722 tablet_bootstrap.cc:654] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:54.059360 15030 tablet_bootstrap.cc:654] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:54.064366 15871 raft_consensus.cc:493] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:57:54.064782 15871 raft_consensus.cc:515] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.065765 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 7588198da5544af99b5fd8e25540c664 (DEFAULT_TABLE table=default.Table_466402e26ef741c8bb19f291f23f54b7 [id=a4ec3a32c3104351bbd088d559e30ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.066399 15871 leader_election.cc:290] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:54.066689 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7588198da5544af99b5fd8e25540c664. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:54.072144 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbdc2eaf4309442888ea49a832da3967 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:54.072530 14931 thread.cc:641] result-tracker (server) Time spent creating pthread: real 0.774s user 0.121s sys 0.181s
W20251211 08:57:54.072866 14931 thread.cc:608] result-tracker (server) Time spent starting thread: real 0.775s user 0.121s sys 0.181s
I20251211 08:57:54.072366 15939 raft_consensus.cc:515] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.074482 15939 leader_election.cc:290] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:54.075177 15952 raft_consensus.cc:493] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:54.075609 15952 raft_consensus.cc:515] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.077507 15952 leader_election.cc:290] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:54.088941 14722 tablet_bootstrap.cc:492] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:54.089478 14722 ts_tablet_manager.cc:1403] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.000s
I20251211 08:57:54.090601 15514 raft_consensus.cc:515] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.091856 14722 raft_consensus.cc:359] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.092562 14722 raft_consensus.cc:385] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:54.092864 14722 raft_consensus.cc:740] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:54.093266 15902 raft_consensus.cc:515] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.094167 15968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.62:39015 every 8 connection(s)
I20251211 08:57:54.095132 14931 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.62:39015
I20251211 08:57:54.095578 15902 leader_election.cc:290] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:54.096022 15514 leader_election.cc:290] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:54.099578 14931 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/master-0/data/info.pb
W20251211 08:57:54.103122 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 47378.8us
W20251211 08:57:54.104230 14542 leader_election.cc:343] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: da57b72207414512b9744b283bc869fe
I20251211 08:57:54.104676 14722 consensus_queue.cc:260] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.105014 14542 leader_election.cc:304] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:54.117305 14260 tablet_service.cc:1505] Processing CreateTablet for tablet 72a704a2d1e043d08ac75bf781e42882 (DEFAULT_TABLE table=default.Table_cd9d0dad407f478f869e3d7654830da1 [id=1ad70c9f427c47c3ad7028c205d96120]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.118463 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a704a2d1e043d08ac75bf781e42882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:54.123854 14722 ts_tablet_manager.cc:1434] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20251211 08:57:54.130640 14722 tablet_bootstrap.cc:492] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:54.149335 15030 tablet_bootstrap.cc:492] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:54.149950 15030 ts_tablet_manager.cc:1403] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.226s user 0.010s sys 0.006s
I20251211 08:57:54.155642 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet ca5eebf2ce3848c79ee2fc21ad2b1486
W20251211 08:57:54.156453 14428 consensus_meta.cc:287] T 020230ff08ed42179a28efbb21613a0f P b276f77f87254a5d909c5ab940a39256: Time spent flushing consensus metadata: real 0.666s user 0.008s sys 0.000s
I20251211 08:57:54.157740 14006 catalog_manager.cc:6724] ReplaceTablet: tablet ca5eebf2ce3848c79ee2fc21ad2b1486 deleted and replaced by tablet 80853e3a1ca24fcdb889aee9dd6619f7
I20251211 08:57:54.158056 14006 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55614 (request call id 526) took 1381 ms. Trace:
I20251211 08:57:54.158298 14006 rpcz_store.cc:276] 1211 08:57:52.776879 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:52.881740 (+104861us) service_pool.cc:225] Handling call
1211 08:57:53.065706 (+183966us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da894f1 00007fc20b6cd227 00007fc20b6cd1b2 00007fc20b6cd0e1 00007fc20b6cd05c 00007fc20b6cc312 00007fc20998455c 00007fc209983996
1211 08:57:54.158005 (+1092299us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.046165 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.064485 (+ 18320us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.064520 (+ 35us) write_op.cc:435] Acquired schema lock
1211 08:57:54.064537 (+ 17us) tablet.cc:660] Decoding operations
1211 08:57:54.064982 (+ 445us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.065035 (+ 53us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.065067 (+ 32us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:54.065402 (+ 335us) tablet.cc:699] Row locks acquired
1211 08:57:54.065421 (+ 19us) write_op.cc:260] PREPARE: finished
1211 08:57:54.065551 (+ 130us) write_op.cc:270] Start()
1211 08:57:54.065646 (+ 95us) write_op.cc:276] Timestamp: P: 1765443474065629 usec, L: 0
1211 08:57:54.065666 (+ 20us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.135172 (+ 69506us) log.cc:844] Serialized 625 byte log entry
1211 08:57:54.152007 (+ 16835us) op_driver.cc:464] REPLICATION: finished
1211 08:57:54.152632 (+ 625us) write_op.cc:301] APPLY: starting
1211 08:57:54.152685 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:54.152783 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:54.152893 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:54.153215 (+ 322us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:54.153425 (+ 210us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:54.153471 (+ 46us) write_op.cc:312] APPLY: finished
1211 08:57:54.153645 (+ 174us) log.cc:844] Serialized 29 byte log entry
1211 08:57:54.153790 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.153856 (+ 66us) write_op.cc:454] Released schema lock
1211 08:57:54.153888 (+ 32us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:54.154131 (+ 243us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.154151 (+ 20us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":419836288,"child_traces":[["op",{"apply.queue_time_us":491,"apply.run_cpu_time_us":1474,"apply.run_wall_time_us":1472,"num_ops":2,"prepare.queue_time_us":622099,"prepare.run_cpu_time_us":2321,"prepare.run_wall_time_us":89283,"raft.queue_time_us":2392,"raft.run_cpu_time_us":591,"raft.run_wall_time_us":591,"replication_time_us":86237}]]}
W20251211 08:57:54.162925 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 132763us
I20251211 08:57:54.163873 14544 leader_election.cc:304] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:54.164775 14261 tablet_service.cc:1552] Processing DeleteTablet for tablet ca5eebf2ce3848c79ee2fc21ad2b1486 with delete_type TABLET_DATA_DELETED (replaced by tablet 80853e3a1ca24fcdb889aee9dd6619f7) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:54.165603 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet ca5eebf2ce3848c79ee2fc21ad2b1486 with delete_type TABLET_DATA_DELETED (replaced by tablet 80853e3a1ca24fcdb889aee9dd6619f7) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:54.168087 14013 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 677ef16a0610477b8ea05233a040cde4 consensus state has no leader
I20251211 08:57:54.168473 14013 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55614 (request call id 527) took 1203 ms. Trace:
I20251211 08:57:54.168699 14013 rpcz_store.cc:276] 1211 08:57:52.965209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.492664 (+527455us) service_pool.cc:225] Handling call
1211 08:57:53.523238 (+ 30574us) spinlock_profiling.cc:243] Waited 29.7 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da87016 00007fc20b6b5b24 00007fc20b6b5ab2 00007fc20b6b59e1 00007fc20b6b595c 00007fc20b6b4c12 00007fc20998455c 00007fc209983996
1211 08:57:53.524148 (+ 910us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:53.524942 (+ 794us) catalog_manager.cc:3708] Apply alter schema
1211 08:57:53.525599 (+ 657us) catalog_manager.cc:3740] Acquired catalog manager lock
1211 08:57:53.525836 (+ 237us) catalog_manager.cc:3898] Updating metadata on disk
1211 08:57:54.167602 (+641766us) catalog_manager.cc:3929] Committing alterations to in-memory state
1211 08:57:54.168426 (+ 824us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.135548 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.135863 (+ 315us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.135882 (+ 19us) write_op.cc:435] Acquired schema lock
1211 08:57:54.135892 (+ 10us) tablet.cc:660] Decoding operations
1211 08:57:54.136207 (+ 315us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.136238 (+ 31us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.136266 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:54.136380 (+ 114us) tablet.cc:699] Row locks acquired
1211 08:57:54.136390 (+ 10us) write_op.cc:260] PREPARE: finished
1211 08:57:54.136476 (+ 86us) write_op.cc:270] Start()
1211 08:57:54.136547 (+ 71us) write_op.cc:276] Timestamp: P: 1765443474136596 usec, L: 0
1211 08:57:54.136560 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.136934 (+ 374us) log.cc:844] Serialized 405 byte log entry
1211 08:57:54.152238 (+ 15304us) op_driver.cc:464] REPLICATION: finished
1211 08:57:54.154373 (+ 2135us) write_op.cc:301] APPLY: starting
1211 08:57:54.154419 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:54.165293 (+ 10874us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:54.165382 (+ 89us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:54.165529 (+ 147us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:54.165633 (+ 104us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:54.165669 (+ 36us) write_op.cc:312] APPLY: finished
1211 08:57:54.165812 (+ 143us) log.cc:844] Serialized 23 byte log entry
1211 08:57:54.165934 (+ 122us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.165983 (+ 49us) write_op.cc:454] Released schema lock
1211 08:57:54.166008 (+ 25us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":68308736,"child_traces":[["op",{"apply.queue_time_us":2023,"apply.run_cpu_time_us":1500,"apply.run_wall_time_us":11788,"num_ops":1,"prepare.queue_time_us":418500,"prepare.run_cpu_time_us":1509,"prepare.run_wall_time_us":1590,"raft.queue_time_us":1362,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":15610,"thread_start_us":14868,"threads_started":1}]]}
I20251211 08:57:53.942624 14905 log.cc:1199] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/7b90b0c944174b2a82696e3fee806934
I20251211 08:57:54.179728 14905 ts_tablet_manager.cc:1950] T 7b90b0c944174b2a82696e3fee806934 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:54.181092 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_2839940578ba47e88ac23a1850c356ed" } modify_external_catalogs: true
I20251211 08:57:54.181597 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_2839940578ba47e88ac23a1850c356ed" } modify_external_catalogs: true
I20251211 08:57:54.185334 14722 tablet_bootstrap.cc:654] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:54.178795 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_752a7e4e23044981af7db22be260121e" } modify_external_catalogs: true
I20251211 08:57:54.198423 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_752a7e4e23044981af7db22be260121e" } modify_external_catalogs: true
W20251211 08:57:54.207150 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 224566us
I20251211 08:57:54.207866 14382 leader_election.cc:304] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:54.223018 14722 tablet_bootstrap.cc:492] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:54.223456 14722 ts_tablet_manager.cc:1403] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.093s user 0.006s sys 0.003s
I20251211 08:57:54.225533 14722 raft_consensus.cc:359] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.226105 14722 raft_consensus.cc:385] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:54.226352 14722 raft_consensus.cc:740] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:54.244982 14722 consensus_queue.cc:260] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.254122 14722 ts_tablet_manager.cc:1434] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.030s user 0.003s sys 0.002s
I20251211 08:57:54.254951 14722 tablet_bootstrap.cc:492] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:54.261801 15936 raft_consensus.cc:493] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:54.262184 15936 raft_consensus.cc:515] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.263938 15936 leader_election.cc:290] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:54.108628 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 14931
I20251211 08:57:54.242627 15969 sys_catalog.cc:263] Verifying existing consensus state
I20251211 08:57:54.275149 14007 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55614 (request call id 529) took 1188 ms. Trace:
I20251211 08:57:54.276358 14007 rpcz_store.cc:276] 1211 08:57:53.086656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.543533 (+456877us) service_pool.cc:225] Handling call
1211 08:57:53.721904 (+178371us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da86996 00007fc20b6ac524 00007fc20b6ac4b2 00007fc20b6ac3e1 00007fc20b6ac35c 00007fc20b6ab612 00007fc20998455c 00007fc209983996
1211 08:57:53.840638 (+118734us) catalog_manager.cc:2101] Acquired catalog manager lock
1211 08:57:53.842532 (+ 1894us) catalog_manager.cc:2147] Created new table and tablet info
1211 08:57:54.274520 (+431988us) catalog_manager.cc:2207] Wrote table and tablets to system table
1211 08:57:54.274817 (+ 297us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1211 08:57:54.275080 (+ 263us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.138702 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.172342 (+ 33640us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.172365 (+ 23us) write_op.cc:435] Acquired schema lock
1211 08:57:54.172376 (+ 11us) tablet.cc:660] Decoding operations
1211 08:57:54.172608 (+ 232us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.172639 (+ 31us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.172657 (+ 18us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:54.172848 (+ 191us) tablet.cc:699] Row locks acquired
1211 08:57:54.172862 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:57:54.172962 (+ 100us) write_op.cc:270] Start()
1211 08:57:54.173034 (+ 72us) write_op.cc:276] Timestamp: P: 1765443474173092 usec, L: 0
1211 08:57:54.173049 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.173536 (+ 487us) log.cc:844] Serialized 433 byte log entry
1211 08:57:54.256776 (+ 83240us) op_driver.cc:464] REPLICATION: finished
1211 08:57:54.257207 (+ 431us) write_op.cc:301] APPLY: starting
1211 08:57:54.257256 (+ 49us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:54.257356 (+ 100us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:54.257463 (+ 107us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:54.257930 (+ 467us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:54.258032 (+ 102us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1211 08:57:54.258067 (+ 35us) write_op.cc:312] APPLY: finished
1211 08:57:54.258187 (+ 120us) log.cc:844] Serialized 29 byte log entry
1211 08:57:54.270095 (+ 11908us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.270165 (+ 70us) write_op.cc:454] Released schema lock
1211 08:57:54.270196 (+ 31us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:54.270439 (+ 243us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.270492 (+ 53us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":408379136,"child_traces":[["op",{"apply.queue_time_us":261,"apply.run_cpu_time_us":1518,"apply.run_wall_time_us":13223,"mutex_wait_us":11717,"num_ops":2,"prepare.queue_time_us":294752,"prepare.run_cpu_time_us":1762,"prepare.run_wall_time_us":34999,"replication_time_us":83646}]]}
I20251211 08:57:54.290503 14012 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55614 (request call id 528) took 1249 ms. Trace:
I20251211 08:57:54.291002 14009 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_000d74366d2342a09dc91f19a5ccdc9f [id=52ae4637827d424eb4d37362bf21cea2] alter complete (version 5)
W20251211 08:57:54.309353 14009 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet f5c5b1aebde94dea81cb37a6a6998070 (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f5c5b1aebde94dea81cb37a6a6998070 consensus state has no leader
W20251211 08:57:54.310632 14009 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 63) took 1258 ms (client timeout 1000 ms). Trace:
I20251211 08:57:54.328588 15969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: Bootstrap starting.
I20251211 08:57:54.332216 14261 tablet_service.cc:1505] Processing CreateTablet for tablet 225438e70aae424a9671aaceb3ee0ed6 (DEFAULT_TABLE table=default.Table_479b5ad717dc450689b1fb6ae89169bf [id=01dc281971114e119a5109457a68e29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.333355 14261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225438e70aae424a9671aaceb3ee0ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:54.333870 14349 heartbeater.cc:344] Connected to a master server at 127.13.146.62:39015
I20251211 08:57:54.340378 14722 tablet_bootstrap.cc:654] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:54.217413 15941 ts_tablet_manager.cc:1929] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:54.346835 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb2ff90b54544420b1aad9d2ab4132ff" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:54.361980 14256 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:46838 (request call id 67) took 1401 ms. Trace:
I20251211 08:57:54.359639 15941 log.cc:1199] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/da52cace2aab4f9dacef18ec5fa56aaf
I20251211 08:57:54.368176 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f785a3cc2e8145eebae7e9c2885b0b1e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:54.370134 14449 raft_consensus.cc:2468] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:54.366931 14256 rpcz_store.cc:276] 1211 08:57:52.960069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:52.961249 (+ 1180us) service_pool.cc:225] Handling call
1211 08:57:54.361881 (+1400632us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:57:54.382582 14714 tablet_bootstrap.cc:654] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:54.390280 15650 raft_consensus.cc:493] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:54.410168 15650 raft_consensus.cc:515] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.411954 15650 leader_election.cc:290] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:54.418220 15696 consensus_meta.cc:287] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256: Time spent flushing consensus metadata: real 0.849s user 0.007s sys 0.010s
I20251211 08:57:54.445947 14231 leader_election.cc:304] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:54.446945 15936 raft_consensus.cc:2804] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:54.447229 15936 raft_consensus.cc:493] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:54.447492 15936 raft_consensus.cc:3060] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.456260 15936 raft_consensus.cc:515] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.458663 15936 leader_election.cc:290] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:54.475975 15696 raft_consensus.cc:515] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.433693 15941 ts_tablet_manager.cc:1950] T da52cace2aab4f9dacef18ec5fa56aaf P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:54.480177 15941 tablet_replica.cc:333] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:54.482282 15941 raft_consensus.cc:2243] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:54.483269 15941 raft_consensus.cc:2272] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:54.439383 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 407740us
W20251211 08:57:54.485491 14385 leader_election.cc:343] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:54.486176 14385 leader_election.cc:304] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:54.490327 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f785a3cc2e8145eebae7e9c2885b0b1e" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:54.491079 14293 raft_consensus.cc:2393] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
I20251211 08:57:54.493561 14293 raft_consensus.cc:3060] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.501916 15977 raft_consensus.cc:2804] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:54.502367 15977 raft_consensus.cc:493] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:54.502848 15977 raft_consensus.cc:3060] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:54.507830 14009 rpcz_store.cc:269] 1211 08:57:53.052546 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.179000 (+126454us) service_pool.cc:225] Handling call
1211 08:57:53.264770 (+ 85770us) spinlock_profiling.cc:243] Waited 82.2 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:54.310571 (+1045801us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.173818 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.174158 (+ 340us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.174176 (+ 18us) write_op.cc:435] Acquired schema lock
1211 08:57:54.174186 (+ 10us) tablet.cc:660] Decoding operations
1211 08:57:54.187665 (+ 13479us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.187696 (+ 31us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.187719 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:54.187842 (+ 123us) tablet.cc:699] Row locks acquired
1211 08:57:54.187856 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:57:54.187951 (+ 95us) write_op.cc:270] Start()
1211 08:57:54.188028 (+ 77us) write_op.cc:276] Timestamp: P: 1765443474188088 usec, L: 0
1211 08:57:54.188045 (+ 17us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.188521 (+ 476us) log.cc:844] Serialized 356 byte log entry
1211 08:57:54.256996 (+ 68475us) op_driver.cc:464] REPLICATION: finished
1211 08:57:54.257591 (+ 595us) write_op.cc:301] APPLY: starting
1211 08:57:54.257634 (+ 43us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:54.257711 (+ 77us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:54.257785 (+ 74us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:54.257908 (+ 123us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:54.258021 (+ 113us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:54.258063 (+ 42us) write_op.cc:312] APPLY: finished
1211 08:57:54.258187 (+ 124us) log.cc:844] Serialized 23 byte log entry
1211 08:57:54.273828 (+ 15641us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.273889 (+ 61us) write_op.cc:454] Released schema lock
1211 08:57:54.273918 (+ 29us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:54.274139 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.274162 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"spinlock_wait_cycles":189116160,"child_traces":[["op",{"apply.queue_time_us":482,"apply.run_cpu_time_us":1084,"apply.run_wall_time_us":16516,"mutex_wait_us":11667,"num_ops":1,"prepare.queue_time_us":292647,"prepare.run_cpu_time_us":1654,"prepare.run_wall_time_us":14883,"replication_time_us":68882}]]}
I20251211 08:57:54.511536 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_481bef424de1464b95f1e738b5986299" } new_table_name: "default.Table_2543626767884e77abfbc8d233ba1db5" modify_external_catalogs: true
I20251211 08:57:54.531093 15991 tablet.cc:1722] T 4b9ae463731d4828afc02039e0d161b5 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:54.531704 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f785a3cc2e8145eebae7e9c2885b0b1e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:54.538370 14447 raft_consensus.cc:3060] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.541837 14904 ts_tablet_manager.cc:1929] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:54.542203 14904 log.cc:1199] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/4ab48f6288494005ad42208e2e97ddf2
I20251211 08:57:54.542829 14904 ts_tablet_manager.cc:1950] T 4ab48f6288494005ad42208e2e97ddf2 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:54.543573 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 393) took 5471 ms. Trace:
I20251211 08:57:54.543860 14904 rpcz_store.cc:276] 1211 08:57:49.072095 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.072524 (+ 429us) service_pool.cc:225] Handling call
1211 08:57:53.139112 (+4066588us) ts_tablet_manager.cc:1151] Deleting tablet 4ab48f6288494005ad42208e2e97ddf2
1211 08:57:53.225172 (+ 86060us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:53.903348 (+678176us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:54.541580 (+638232us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:54.543523 (+ 1943us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4064688}
I20251211 08:57:54.533130 14443 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:54.547219 14443 raft_consensus.cc:2468] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:54.548874 14447 raft_consensus.cc:2468] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:54.563992 15939 raft_consensus.cc:2804] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:54.564518 15939 raft_consensus.cc:697] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:54.565351 15939 consensus_queue.cc:237] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.566612 15030 raft_consensus.cc:359] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:54.566962 15871 raft_consensus.cc:493] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:54.567306 15030 raft_consensus.cc:385] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:54.567587 15030 raft_consensus.cc:740] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:54.567543 15871 raft_consensus.cc:515] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.568341 15030 consensus_queue.cc:260] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:54.568734 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 350) took 8997 ms. Trace:
I20251211 08:57:54.567023 15940 raft_consensus.cc:493] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:54.569383 15871 leader_election.cc:290] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:54.569772 15871 raft_consensus.cc:493] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:54.569558 15940 raft_consensus.cc:515] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.570261 15030 ts_tablet_manager.cc:1434] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.420s user 0.004s sys 0.000s
I20251211 08:57:54.570137 15871 raft_consensus.cc:515] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:54.570894 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 407259us
I20251211 08:57:54.571314 15940 leader_election.cc:290] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:54.571979 15940 raft_consensus.cc:2804] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:54.574082 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f785a3cc2e8145eebae7e9c2885b0b1e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:54.574106 14230 leader_election.cc:304] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:54.574576 14608 raft_consensus.cc:2468] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:54.574918 15950 raft_consensus.cc:3060] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.577098 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da57b72207414512b9744b283bc869fe" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:54.577462 14601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:54.577741 14604 raft_consensus.cc:2368] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b for earlier term 1. Current term is 2.
I20251211 08:57:54.578066 14601 raft_consensus.cc:2468] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:54.578958 14602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f785a3cc2e8145eebae7e9c2885b0b1e" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:54.579535 14602 raft_consensus.cc:3060] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.580832 15899 raft_consensus.cc:493] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:54.581247 15899 raft_consensus.cc:515] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.583493 14655 heartbeater.cc:344] Connected to a master server at 127.13.146.62:39015
I20251211 08:57:54.585433 15899 leader_election.cc:290] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:54.586167 15871 leader_election.cc:290] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:54.589589 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45c7d841bd3643a799a8b96e82d6d675" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:54.590202 14296 raft_consensus.cc:2468] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:54.591101 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da57b72207414512b9744b283bc869fe" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:54.591640 14295 raft_consensus.cc:3060] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 2
I20251211 08:57:54.586184 15030 tablet_bootstrap.cc:492] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:54.593657 14601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:54.594244 14601 raft_consensus.cc:2468] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:54.595191 14294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e5c4bf71b44abca66b0e94995c533e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:54.596127 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88860fa9d7e6414185f0a1cbc78b2571" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:54.596603 14601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e05628011634149b17d298c54074d00" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:54.597172 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e15483f539f24f1ba1b442663d66fd45" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:54.597602 14299 raft_consensus.cc:3060] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:57:54.598477 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6d0fef50524476dbd47fc70e4bda319" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:54.599275 14607 raft_consensus.cc:2393] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:54.600677 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341b762afd8440bb7198ca134c8dffe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:54.601099 14604 raft_consensus.cc:3060] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.603094 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da52cace2aab4f9dacef18ec5fa56aaf" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:54.605790 14230 leader_election.cc:304] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:54.605968 14602 raft_consensus.cc:2468] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:54.583514 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 72a704a2d1e043d08ac75bf781e42882 (DEFAULT_TABLE table=default.Table_cd9d0dad407f478f869e3d7654830da1 [id=1ad70c9f427c47c3ad7028c205d96120]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.607309 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f785a3cc2e8145eebae7e9c2885b0b1e" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:54.608263 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a704a2d1e043d08ac75bf781e42882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:54.608557 14607 raft_consensus.cc:2393] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
I20251211 08:57:54.609584 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 225438e70aae424a9671aaceb3ee0ed6 (DEFAULT_TABLE table=default.Table_479b5ad717dc450689b1fb6ae89169bf [id=01dc281971114e119a5109457a68e29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.610582 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225438e70aae424a9671aaceb3ee0ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:54.612625 14230 leader_election.cc:304] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:54.568994 14905 rpcz_store.cc:276] 1211 08:57:45.571515 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.571650 (+ 135us) service_pool.cc:225] Handling call
1211 08:57:53.860967 (+8289317us) ts_tablet_manager.cc:1151] Deleting tablet 7b90b0c944174b2a82696e3fee806934
1211 08:57:53.861018 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:53.931693 (+ 70675us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:53.942130 (+ 10437us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:54.568684 (+626554us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":401989,"tablet-delete.queue_time_us":8288796}
I20251211 08:57:54.614087 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 351) took 8807 ms. Trace:
I20251211 08:57:54.614392 14905 rpcz_store.cc:276] 1211 08:57:45.807151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:45.807370 (+ 219us) service_pool.cc:225] Handling call
1211 08:57:54.613690 (+8806320us) ts_tablet_manager.cc:1151] Deleting tablet 6874fb018bc844b3aeeab7db8fc53e9a
1211 08:57:54.613748 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:54.614035 (+ 287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8805746}
I20251211 08:57:54.615419 14905 tablet_replica.cc:333] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:54.616132 14905 raft_consensus.cc:2243] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:54.616685 14905 raft_consensus.cc:2272] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:54.618508 14604 raft_consensus.cc:2468] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:54.625553 14295 raft_consensus.cc:2468] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 2.
I20251211 08:57:54.625710 14382 leader_election.cc:304] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:54.572845 15940 raft_consensus.cc:493] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:54.626617 15940 raft_consensus.cc:3060] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.627182 15944 raft_consensus.cc:2804] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:54.627632 15944 raft_consensus.cc:493] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:54.628691 15944 raft_consensus.cc:3060] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.646428 14905 ts_tablet_manager.cc:1916] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:54.652760 15030 tablet_bootstrap.cc:654] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:54.639456 15989 tablet.cc:1722] T 4b9ae463731d4828afc02039e0d161b5 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:54.643728 14299 raft_consensus.cc:2468] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 2.
I20251211 08:57:54.659180 15952 raft_consensus.cc:2764] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251211 08:57:54.644531 15950 raft_consensus.cc:2749] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:54.662935 15942 raft_consensus.cc:2749] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:54.478045 15696 leader_election.cc:290] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:54.672796 14447 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e05628011634149b17d298c54074d00" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:54.678135 14443 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e5c4bf71b44abca66b0e94995c533e" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:54.678637 14443 raft_consensus.cc:3060] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.680960 14714 tablet_bootstrap.cc:492] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:54.681389 14714 ts_tablet_manager.cc:1403] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.811s user 0.013s sys 0.005s
I20251211 08:57:54.683512 14714 raft_consensus.cc:359] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:54.647867 14722 tablet_bootstrap.cc:492] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:54.650195 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341b762afd8440bb7198ca134c8dffe" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:54.687789 14722 ts_tablet_manager.cc:1403] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.433s user 0.016s sys 0.029s
W20251211 08:57:54.688377 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 75865.6us
I20251211 08:57:54.690375 14722 raft_consensus.cc:359] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.690943 14722 raft_consensus.cc:385] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:54.691031 14230 leader_election.cc:400] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Vote denied by peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) with higher term. Message: Invalid argument: T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b for earlier term 1. Current term is 2.
I20251211 08:57:54.650374 15997 raft_consensus.cc:2749] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:54.691669 14905 ts_tablet_manager.cc:1929] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:54.692065 14905 log.cc:1199] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/14fc87cb76024b5b8227dfe9939e1c89
I20251211 08:57:54.692713 14905 ts_tablet_manager.cc:1950] T 14fc87cb76024b5b8227dfe9939e1c89 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:54.634514 15940 raft_consensus.cc:515] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.693447 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 353) took 8524 ms. Trace:
I20251211 08:57:54.693977 14601 raft_consensus.cc:2393] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate c5e624bcfd154cd586ef1dedb9940063 in this term.
I20251211 08:57:54.693732 14905 rpcz_store.cc:276] 1211 08:57:46.168964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.169629 (+ 665us) service_pool.cc:225] Handling call
1211 08:57:54.615251 (+8445622us) ts_tablet_manager.cc:1151] Deleting tablet 14fc87cb76024b5b8227dfe9939e1c89
1211 08:57:54.615309 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:54.681952 (+ 66643us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:54.691458 (+ 9506us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:54.693400 (+ 1942us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8444928}
I20251211 08:57:54.696038 15940 leader_election.cc:290] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:54.703833 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 357) took 8328 ms. Trace:
I20251211 08:57:54.691202 14722 raft_consensus.cc:740] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:54.691334 14230 leader_election.cc:403] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20251211 08:57:54.704322 14905 rpcz_store.cc:276] 1211 08:57:46.374934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.375403 (+ 469us) service_pool.cc:225] Handling call
1211 08:57:54.695125 (+8319722us) ts_tablet_manager.cc:1151] Deleting tablet c6d419632b2a49e1975d4baa8ac85322
1211 08:57:54.703364 (+ 8239us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:54.703760 (+ 396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8319232}
I20251211 08:57:54.705173 15996 raft_consensus.cc:2749] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) with higher term. Message: Invalid argument: T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b for earlier term 1. Current term is 2.
I20251211 08:57:54.706530 14230 leader_election.cc:304] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:54.707103 15996 raft_consensus.cc:2804] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:54.707374 15996 raft_consensus.cc:493] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:54.707617 15996 raft_consensus.cc:3060] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.712850 14905 tablet_replica.cc:333] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:54.706903 14722 consensus_queue.cc:260] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.713654 14905 raft_consensus.cc:2243] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:54.714265 14905 raft_consensus.cc:2272] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:54.714241 15996 raft_consensus.cc:515] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.714712 14722 ts_tablet_manager.cc:1434] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20251211 08:57:54.715801 14905 ts_tablet_manager.cc:1916] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:54.716202 15996 leader_election.cc:290] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:54.716776 14230 leader_election.cc:304] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:54.717609 14230 leader_election.cc:304] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:54.718457 15950 raft_consensus.cc:2804] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:54.718696 14544 leader_election.cc:304] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:54.718973 15950 raft_consensus.cc:697] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
W20251211 08:57:54.719601 14544 leader_election.cc:343] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:54.719676 15950 consensus_queue.cc:237] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:54.720168 14544 leader_election.cc:343] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:54.720845 14544 leader_election.cc:304] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:54.721524 14544 leader_election.cc:304] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:54.722333 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6d0fef50524476dbd47fc70e4bda319" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:54.723177 15940 raft_consensus.cc:2804] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:57:54.723574 15940 raft_consensus.cc:493] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:57:54.723960 15940 raft_consensus.cc:3060] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:57:54.724658 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e05628011634149b17d298c54074d00" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:54.725128 14295 raft_consensus.cc:3060] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.726286 14294 raft_consensus.cc:3060] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.727059 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:54.727808 14297 raft_consensus.cc:2393] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5e624bcfd154cd586ef1dedb9940063 in current term 1: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
I20251211 08:57:54.728654 15939 raft_consensus.cc:2804] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader election won for term 2
I20251211 08:57:54.729437 15964 raft_consensus.cc:2804] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader election won for term 2
I20251211 08:57:54.729842 15964 raft_consensus.cc:697] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:54.731208 14295 raft_consensus.cc:2468] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:54.730721 15964 consensus_queue.cc:237] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.718349 15997 raft_consensus.cc:3060] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.735163 15939 raft_consensus.cc:697] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:54.736014 15939 consensus_queue.cc:237] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.747135 14544 leader_election.cc:304] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:54.747948 15871 raft_consensus.cc:2804] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:54.748631 15871 raft_consensus.cc:697] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:54.749516 15871 consensus_queue.cc:237] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.757167 15997 raft_consensus.cc:2749] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:54.757844 14293 raft_consensus.cc:2468] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:54.759527 15902 consensus_queue.cc:1048] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:54.762264 14544 leader_election.cc:304] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:54.765136 15902 raft_consensus.cc:2804] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:57:54.765594 15902 raft_consensus.cc:493] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:54.766093 15902 raft_consensus.cc:3060] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:57:54.768748 15952 raft_consensus.cc:493] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:54.769124 15952 raft_consensus.cc:515] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.770881 15952 leader_election.cc:290] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:54.775370 15989 tablet.cc:1722] T e354416047f74d02a9455147ca8d6bf3 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:54.797706 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:54.798316 14606 raft_consensus.cc:3060] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:54.466166 14004 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55614 (request call id 531) took 1081 ms. Trace:
I20251211 08:57:54.466895 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet b49ea29851d94622b4299e93804a0578
I20251211 08:57:54.809090 14006 master_service.cc:954] ReplaceTablet: received request to replace tablet f2dc6cb4efc744e6904728ad120df934 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:54.301525 14012 rpcz_store.cc:276] 1211 08:57:53.040861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.517066 (+476205us) service_pool.cc:225] Handling call
1211 08:57:53.726671 (+209605us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da87016 00007fc20b6b5b24 00007fc20b6b5ab2 00007fc20b6b59e1 00007fc20b6b595c 00007fc20b6b4c12 00007fc20998455c 00007fc209983996
1211 08:57:53.859032 (+132361us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:53.859991 (+ 959us) catalog_manager.cc:3708] Apply alter schema
1211 08:57:53.860771 (+ 780us) catalog_manager.cc:3740] Acquired catalog manager lock
1211 08:57:53.881266 (+ 20495us) catalog_manager.cc:3898] Updating metadata on disk
1211 08:57:54.282939 (+401673us) catalog_manager.cc:3929] Committing alterations to in-memory state
1211 08:57:54.290447 (+ 7508us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.201543 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.201861 (+ 318us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.201879 (+ 18us) write_op.cc:435] Acquired schema lock
1211 08:57:54.201889 (+ 10us) tablet.cc:660] Decoding operations
1211 08:57:54.202080 (+ 191us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.202117 (+ 37us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.202147 (+ 30us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:54.202307 (+ 160us) tablet.cc:699] Row locks acquired
1211 08:57:54.202323 (+ 16us) write_op.cc:260] PREPARE: finished
1211 08:57:54.217821 (+ 15498us) write_op.cc:270] Start()
1211 08:57:54.217930 (+ 109us) write_op.cc:276] Timestamp: P: 1765443474217949 usec, L: 0
1211 08:57:54.217969 (+ 39us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.227326 (+ 9357us) log.cc:844] Serialized 405 byte log entry
1211 08:57:54.258796 (+ 31470us) op_driver.cc:464] REPLICATION: finished
1211 08:57:54.270730 (+ 11934us) write_op.cc:301] APPLY: starting
1211 08:57:54.270783 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:54.270888 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:54.270994 (+ 106us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:54.271141 (+ 147us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:54.271249 (+ 108us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:54.271282 (+ 33us) write_op.cc:312] APPLY: finished
1211 08:57:54.271435 (+ 153us) log.cc:844] Serialized 23 byte log entry
1211 08:57:54.271544 (+ 109us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.271596 (+ 52us) write_op.cc:454] Released schema lock
1211 08:57:54.271622 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":480475008,"child_traces":[["op",{"apply.queue_time_us":11808,"apply.run_cpu_time_us":1045,"apply.run_wall_time_us":1044,"num_ops":1,"prepare.queue_time_us":213857,"prepare.run_cpu_time_us":1853,"prepare.run_wall_time_us":25944,"replication_time_us":40722,"threads_started":1}]]}
I20251211 08:57:54.523203 14005 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55614 (request call id 530) took 1221 ms. Trace:
I20251211 08:57:54.811203 14606 raft_consensus.cc:2468] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:54.814177 14905 ts_tablet_manager.cc:1929] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:54.814591 14905 log.cc:1199] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/00e5c4bf71b44abca66b0e94995c533e
I20251211 08:57:54.545163 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 4ab48f6288494005ad42208e2e97ddf2 (table default.Table_cd9ec56dc08c48779e88a778d2917bb0 [id=658ce49773af41519ab660f3bc29975b]) successfully deleted
I20251211 08:57:54.574437 14007 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 4b9ae463731d4828afc02039e0d161b5 (table default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]). Expected version 1 got 0
W20251211 08:57:54.814953 14000 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 269918us
I20251211 08:57:54.815346 14007 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:57:54.816655 14007 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_31fcfbed4a2c4e8aa05fb42c261bf1d5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 2 got 0
I20251211 08:57:54.818231 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet b49ea29851d94622b4299e93804a0578 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:54 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:54.650857 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet da52cace2aab4f9dacef18ec5fa56aaf (table default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]) successfully deleted
W20251211 08:57:54.819006 13998 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 168246us
I20251211 08:57:54.820001 14905 ts_tablet_manager.cc:1950] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:57:54.820039 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 98d14ba15dcd47f8b9e8035b3c7d477c because the tablet was not found. No further retry: Not found: Tablet not found: 98d14ba15dcd47f8b9e8035b3c7d477c
W20251211 08:57:54.820590 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 45c7d841bd3643a799a8b96e82d6d675 because the tablet was not found. No further retry: Not found: Tablet not found: 45c7d841bd3643a799a8b96e82d6d675
I20251211 08:57:54.821710 14260 tablet_service.cc:1505] Processing CreateTablet for tablet 7564d643467744709032605d73480b56 (DEFAULT_TABLE table=default.Table_a522cb7af78a49aaaa070d2ee83bdb19 [id=b5f79325e5de44c8a049f137640b617f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.619359 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 7b90b0c944174b2a82696e3fee806934 (table default.Table_e3e740f7ffee4b1683b5888496b1a384 [id=0f8c030be5504d859ef25176b6cbb4b1]) successfully deleted
W20251211 08:57:54.824821 13997 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 205579us
I20251211 08:57:54.804329 14004 rpcz_store.cc:276] 1211 08:57:53.384753 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.855330 (+470577us) service_pool.cc:225] Handling call
1211 08:57:53.991155 (+135825us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da86996 00007fc20b6ac524 00007fc20b6ac4b2 00007fc20b6ac3e1 00007fc20b6ac35c 00007fc20b6ab612 00007fc20998455c 00007fc209983996
1211 08:57:53.993274 (+ 2119us) catalog_manager.cc:2101] Acquired catalog manager lock
1211 08:57:54.136248 (+142974us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b0c000345e8. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc204a1a781 00007fc204b279df 00007fc204b27819 00007fc20d96763f 00007fc20d9652f1 00007fc20d9629c7 00007fc20d9664ce 00007fc20da869cf 00007fc20b6ac524 00007fc20b6ac4b2 00007fc20b6ac3e1
1211 08:57:54.136828 (+ 580us) catalog_manager.cc:2147] Created new table and tablet info
1211 08:57:54.465561 (+328733us) catalog_manager.cc:2207] Wrote table and tablets to system table
1211 08:57:54.465882 (+ 321us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1211 08:57:54.466111 (+ 229us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.227594 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.227891 (+ 297us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.227910 (+ 19us) write_op.cc:435] Acquired schema lock
1211 08:57:54.227919 (+ 9us) tablet.cc:660] Decoding operations
1211 08:57:54.228163 (+ 244us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.228208 (+ 45us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.228251 (+ 43us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:54.228466 (+ 215us) tablet.cc:699] Row locks acquired
1211 08:57:54.228484 (+ 18us) write_op.cc:260] PREPARE: finished
1211 08:57:54.228568 (+ 84us) write_op.cc:270] Start()
1211 08:57:54.228632 (+ 64us) write_op.cc:276] Timestamp: P: 1765443474228726 usec, L: 0
1211 08:57:54.228646 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.229020 (+ 374us) log.cc:844] Serialized 433 byte log entry
1211 08:57:54.460852 (+231832us) op_driver.cc:464] REPLICATION: finished
1211 08:57:54.461307 (+ 455us) write_op.cc:301] APPLY: starting
1211 08:57:54.461376 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:54.461491 (+ 115us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:54.461596 (+ 105us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:54.462044 (+ 448us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:54.462208 (+ 164us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1211 08:57:54.462264 (+ 56us) write_op.cc:312] APPLY: finished
1211 08:57:54.462486 (+ 222us) log.cc:844] Serialized 29 byte log entry
1211 08:57:54.462635 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.462709 (+ 74us) write_op.cc:454] Released schema lock
1211 08:57:54.462736 (+ 27us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:54.462985 (+ 249us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.463008 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":634471936,"child_traces":[["op",{"apply.queue_time_us":291,"apply.run_cpu_time_us":1645,"apply.run_wall_time_us":1644,"num_ops":2,"prepare.queue_time_us":87382,"prepare.run_cpu_time_us":1576,"prepare.run_wall_time_us":1573,"replication_time_us":232100}]]}
I20251211 08:57:54.825834 14584 tablet_service.cc:1552] Processing DeleteTablet for tablet b49ea29851d94622b4299e93804a0578 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:54 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:54.827371 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a6d23dddea0d46189b02672fbc0c3d04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:54.829304 16008 tablet.cc:1722] T 4b9ae463731d4828afc02039e0d161b5 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:54.812407 14005 rpcz_store.cc:276] 1211 08:57:53.301526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.670086 (+368560us) service_pool.cc:225] Handling call
1211 08:57:53.725398 (+ 55312us) spinlock_profiling.cc:243] Waited 52.3 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da86996 00007fc20b6ac524 00007fc20b6ac4b2 00007fc20b6ac3e1 00007fc20b6ac35c 00007fc20b6ab612 00007fc20998455c 00007fc209983996
1211 08:57:53.846023 (+120625us) catalog_manager.cc:2101] Acquired catalog manager lock
1211 08:57:53.858406 (+ 12383us) spinlock_profiling.cc:243] Waited 9.62 ms on lock 0x7b0c000345e8. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc204a1a781 00007fc204b279df 00007fc204b27819 00007fc20d96763f 00007fc20d9652f1 00007fc20d9629c7 00007fc20d9664ce 00007fc20da869cf 00007fc20b6ac524 00007fc20b6ac4b2 00007fc20b6ac3e1
1211 08:57:53.862557 (+ 4151us) catalog_manager.cc:2147] Created new table and tablet info
1211 08:57:54.522533 (+659976us) catalog_manager.cc:2207] Wrote table and tablets to system table
1211 08:57:54.522905 (+ 372us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1211 08:57:54.523148 (+ 243us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.188812 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.189144 (+ 332us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.189166 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:57:54.189179 (+ 13us) tablet.cc:660] Decoding operations
1211 08:57:54.200560 (+ 11381us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.200591 (+ 31us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.200610 (+ 19us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:54.200756 (+ 146us) tablet.cc:699] Row locks acquired
1211 08:57:54.200767 (+ 11us) write_op.cc:260] PREPARE: finished
1211 08:57:54.200854 (+ 87us) write_op.cc:270] Start()
1211 08:57:54.200919 (+ 65us) write_op.cc:276] Timestamp: P: 1765443474200997 usec, L: 0
1211 08:57:54.200933 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.201295 (+ 362us) log.cc:844] Serialized 433 byte log entry
1211 08:57:54.257187 (+ 55892us) op_driver.cc:464] REPLICATION: finished
1211 08:57:54.258891 (+ 1704us) write_op.cc:301] APPLY: starting
1211 08:57:54.258938 (+ 47us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:54.259027 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:54.259187 (+ 160us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:54.259713 (+ 526us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:54.259841 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1211 08:57:54.259877 (+ 36us) write_op.cc:312] APPLY: finished
1211 08:57:54.260033 (+ 156us) log.cc:844] Serialized 29 byte log entry
1211 08:57:54.522231 (+262198us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.522299 (+ 68us) write_op.cc:454] Released schema lock
1211 08:57:54.522341 (+ 42us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:54.523624 (+ 1283us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.523648 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":142483712,"child_traces":[["op",{"apply.queue_time_us":1563,"apply.run_cpu_time_us":1706,"apply.run_wall_time_us":264707,"mutex_wait_us":261995,"num_ops":2,"prepare.queue_time_us":202406,"prepare.run_cpu_time_us":1555,"prepare.run_wall_time_us":12647,"replication_time_us":56183,"thread_start_us":1300,"threads_started":1}]]}
W20251211 08:57:54.839646 14542 leader_election.cc:343] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 45c7d841bd3643a799a8b96e82d6d675
W20251211 08:57:54.841249 14542 leader_election.cc:343] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: e15483f539f24f1ba1b442663d66fd45
W20251211 08:57:54.842022 14542 leader_election.cc:343] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: da57b72207414512b9744b283bc869fe
W20251211 08:57:54.825824 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 6874fb018bc844b3aeeab7db8fc53e9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6874fb018bc844b3aeeab7db8fc53e9a already in progress: creating tablet
I20251211 08:57:54.842912 15899 consensus_queue.cc:1048] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:54.843369 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 14fc87cb76024b5b8227dfe9939e1c89 (table default.Table_fd2daf1c25534c25b34cf363453b0e67 [id=6e2d344612214ee382863680435345b7]) successfully deleted
W20251211 08:57:54.844022 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet c6d419632b2a49e1975d4baa8ac85322 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6d419632b2a49e1975d4baa8ac85322 already in progress: creating tablet
W20251211 08:57:54.845042 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6d23dddea0d46189b02672fbc0c3d04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:54.845263 14542 leader_election.cc:343] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: da52cace2aab4f9dacef18ec5fa56aaf
I20251211 08:57:54.845662 14542 leader_election.cc:304] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:54.848810 15940 raft_consensus.cc:515] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.853335 15997 raft_consensus.cc:493] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:54.853824 15997 raft_consensus.cc:515] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.855705 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 358) took 8480 ms. Trace:
I20251211 08:57:54.855891 15997 leader_election.cc:290] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:54.856036 14905 rpcz_store.cc:276] 1211 08:57:46.375286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.376478 (+ 1192us) service_pool.cc:225] Handling call
1211 08:57:54.708833 (+8332355us) ts_tablet_manager.cc:1151] Deleting tablet 00e5c4bf71b44abca66b0e94995c533e
1211 08:57:54.712746 (+ 3913us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:54.752734 (+ 39988us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:54.813894 (+ 61160us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:54.855628 (+ 41734us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8331904}
I20251211 08:57:54.857517 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 365) took 8305 ms. Trace:
I20251211 08:57:54.857744 14905 rpcz_store.cc:276] 1211 08:57:46.552200 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.552771 (+ 571us) service_pool.cc:225] Handling call
1211 08:57:54.857184 (+8304413us) ts_tablet_manager.cc:1151] Deleting tablet faa96580439242948a66bbf00aeb836a
1211 08:57:54.857230 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:54.857477 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8303691}
I20251211 08:57:54.859668 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 00e5c4bf71b44abca66b0e94995c533e (table default.Table_dcb6d015c5c543689fc117f18bdff260 [id=ff5165bdae864e59a46b71d1ca8d8611]) successfully deleted
W20251211 08:57:54.860417 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet faa96580439242948a66bbf00aeb836a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faa96580439242948a66bbf00aeb836a already in progress: creating tablet
I20251211 08:57:54.858624 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 368) took 8128 ms. Trace:
I20251211 08:57:54.861346 14905 rpcz_store.cc:276] 1211 08:57:46.730376 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.737607 (+ 7231us) service_pool.cc:225] Handling call
1211 08:57:54.858325 (+8120718us) ts_tablet_manager.cc:1151] Deleting tablet 9821822304c948f3bfcf5d52901371a4
1211 08:57:54.858373 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:54.858588 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7973990}
I20251211 08:57:54.862707 14905 tablet_replica.cc:333] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:54.863596 14905 raft_consensus.cc:2243] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251211 08:57:54.864127 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 9821822304c948f3bfcf5d52901371a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9821822304c948f3bfcf5d52901371a4 already in progress: creating tablet
I20251211 08:57:54.865001 14230 leader_election.cc:304] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:54.865862 15997 raft_consensus.cc:2804] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:54.866250 15997 raft_consensus.cc:697] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:54.866940 15997 consensus_queue.cc:237] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.868031 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 7564d643467744709032605d73480b56 (DEFAULT_TABLE table=default.Table_a522cb7af78a49aaaa070d2ee83bdb19 [id=b5f79325e5de44c8a049f137640b617f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.869019 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7564d643467744709032605d73480b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:54.884733 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45c7d841bd3643a799a8b96e82d6d675" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:54.885214 14293 raft_consensus.cc:3060] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 2
I20251211 08:57:54.892279 14293 raft_consensus.cc:2468] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 2.
W20251211 08:57:54.901257 14231 leader_election.cc:343] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: dd14de369e084828afb8a924c46c649b
W20251211 08:57:54.902331 14353 tablet.cc:2378] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251211 08:57:54.921290 14722 tablet_bootstrap.cc:492] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:54.923494 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7564d643467744709032605d73480b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:54.929172 15902 raft_consensus.cc:515] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.937775 14722 tablet_bootstrap.cc:654] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:54.942416 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 7f0cd7bb9fae4a2393578922fd359e6e (table default.Table_10d2cf5a95b9442da2fb73a50e9cbd8d [id=1e8b5ddeeea54c5f8e1951c184d77c27], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7f0cd7bb9fae4a2393578922fd359e6e consensus state has no leader
I20251211 08:57:54.945333 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 395) took 5838 ms. Trace:
I20251211 08:57:54.945724 14904 rpcz_store.cc:276] 1211 08:57:49.107267 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.119953 (+ 12686us) service_pool.cc:225] Handling call
1211 08:57:54.865646 (+5745693us) ts_tablet_manager.cc:1151] Deleting tablet f2f095b59ecf44ffabaa0ac22212938b
1211 08:57:54.945035 (+ 79389us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:54.945264 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5730696}
W20251211 08:57:54.947196 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet f2f095b59ecf44ffabaa0ac22212938b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2f095b59ecf44ffabaa0ac22212938b already in progress: creating tablet
I20251211 08:57:54.948040 14904 tablet_replica.cc:333] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:54.949081 14904 raft_consensus.cc:2243] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:54.974176 14714 raft_consensus.cc:385] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:54.977577 14714 raft_consensus.cc:740] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:54.977816 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 143f20f5c37948068ff2fb0d92f0f34b
I20251211 08:57:54.975044 14010 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55614 (request call id 532) took 1228 ms. Trace:
I20251211 08:57:54.979174 14714 consensus_queue.cc:260] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:54.981115 14714 ts_tablet_manager.cc:1434] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.299s user 0.003s sys 0.000s
I20251211 08:57:54.979280 14010 rpcz_store.cc:276] 1211 08:57:53.746985 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.908621 (+161636us) service_pool.cc:225] Handling call
1211 08:57:53.996579 (+ 87958us) spinlock_profiling.cc:243] Waited 87.3 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da86996 00007fc20b6ac524 00007fc20b6ac4b2 00007fc20b6ac3e1 00007fc20b6ac35c 00007fc20b6ab612 00007fc20998455c 00007fc209983996
1211 08:57:54.072648 (+ 76069us) catalog_manager.cc:2101] Acquired catalog manager lock
1211 08:57:54.227175 (+154527us) catalog_manager.cc:2147] Created new table and tablet info
1211 08:57:54.974462 (+747287us) catalog_manager.cc:2207] Wrote table and tablets to system table
1211 08:57:54.974736 (+ 274us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1211 08:57:54.974992 (+ 256us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.471860 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.472220 (+ 360us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.472240 (+ 20us) write_op.cc:435] Acquired schema lock
1211 08:57:54.472251 (+ 11us) tablet.cc:660] Decoding operations
1211 08:57:54.472490 (+ 239us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.472521 (+ 31us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.472541 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:54.472698 (+ 157us) tablet.cc:699] Row locks acquired
1211 08:57:54.472728 (+ 30us) write_op.cc:260] PREPARE: finished
1211 08:57:54.472822 (+ 94us) write_op.cc:270] Start()
1211 08:57:54.472891 (+ 69us) write_op.cc:276] Timestamp: P: 1765443474472864 usec, L: 0
1211 08:57:54.472905 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.473308 (+ 403us) log.cc:844] Serialized 433 byte log entry
1211 08:57:54.969773 (+496465us) op_driver.cc:464] REPLICATION: finished
1211 08:57:54.971621 (+ 1848us) write_op.cc:301] APPLY: starting
1211 08:57:54.971668 (+ 47us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:54.971746 (+ 78us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:54.971820 (+ 74us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:54.972231 (+ 411us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:54.972331 (+ 100us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1211 08:57:54.972361 (+ 30us) write_op.cc:312] APPLY: finished
1211 08:57:54.972511 (+ 150us) log.cc:844] Serialized 29 byte log entry
1211 08:57:54.972629 (+ 118us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.972682 (+ 53us) write_op.cc:454] Released schema lock
1211 08:57:54.972707 (+ 25us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:54.972898 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.972921 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":200739584,"child_traces":[["op",{"apply.queue_time_us":1743,"apply.run_cpu_time_us":1242,"apply.run_wall_time_us":1241,"num_ops":2,"prepare.queue_time_us":68,"prepare.run_cpu_time_us":1619,"prepare.run_wall_time_us":1618,"replication_time_us":496802}]]}
I20251211 08:57:54.981874 14714 tablet_bootstrap.cc:492] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:54.684410 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 58799.7us
I20251211 08:57:54.684837 14443 raft_consensus.cc:2468] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:54.686714 14444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88860fa9d7e6414185f0a1cbc78b2571" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:54.986997 14444 raft_consensus.cc:2468] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:54.961040 14261 tablet_service.cc:1505] Processing CreateTablet for tablet 715049b6a9214f48a0b573ac4f296424 (DEFAULT_TABLE table=default.Table_bd5960f04a874bee8826f8d460f1388b [id=922194492b5f427aaa346a3ad93b2d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.990371 14261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 715049b6a9214f48a0b573ac4f296424. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:54.992086 14382 leader_election.cc:304] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:54.992806 15650 raft_consensus.cc:2749] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:54.685514 15944 raft_consensus.cc:515] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:54.688946 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 7588198da5544af99b5fd8e25540c664 (DEFAULT_TABLE table=default.Table_466402e26ef741c8bb19f291f23f54b7 [id=a4ec3a32c3104351bbd088d559e30ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.995539 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7588198da5544af99b5fd8e25540c664. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:54.689852 14417 tablet_service.cc:1552] Processing DeleteTablet for tablet ca5eebf2ce3848c79ee2fc21ad2b1486 with delete_type TABLET_DATA_DELETED (replaced by tablet 80853e3a1ca24fcdb889aee9dd6619f7) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:54.712020 14445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:54.735385 14446 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e05628011634149b17d298c54074d00" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:54.779048 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:54.802954 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 72a704a2d1e043d08ac75bf781e42882 (DEFAULT_TABLE table=default.Table_cd9d0dad407f478f869e3d7654830da1 [id=1ad70c9f427c47c3ad7028c205d96120]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.803161 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 225438e70aae424a9671aaceb3ee0ed6 (DEFAULT_TABLE table=default.Table_479b5ad717dc450689b1fb6ae89169bf [id=01dc281971114e119a5109457a68e29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.803576 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 7564d643467744709032605d73480b56 (DEFAULT_TABLE table=default.Table_a522cb7af78a49aaaa070d2ee83bdb19 [id=b5f79325e5de44c8a049f137640b617f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:54.833050 15941 ts_tablet_manager.cc:1916] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:54.862903 15991 tablet_replica.cc:1410] Time spent flushing the Log queue took a long time: real 0.313s user 0.000s sys 0.001s
I20251211 08:57:55.001091 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a704a2d1e043d08ac75bf781e42882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.003309 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225438e70aae424a9671aaceb3ee0ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.003782 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7564d643467744709032605d73480b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.019166 14904 raft_consensus.cc:2272] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:55.019843 14445 raft_consensus.cc:2393] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5e624bcfd154cd586ef1dedb9940063 in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:55.021088 14904 ts_tablet_manager.cc:1916] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:55.022977 15944 leader_election.cc:290] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:55.027432 14722 tablet_bootstrap.cc:492] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:55.027806 14722 ts_tablet_manager.cc:1403] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.107s user 0.008s sys 0.001s
I20251211 08:57:55.029960 14722 raft_consensus.cc:359] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:55.030555 14722 raft_consensus.cc:385] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:54.895009 16007 tablet.cc:1722] T e354416047f74d02a9455147ca8d6bf3 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:54.904937 14442 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341b762afd8440bb7198ca134c8dffe" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:54.933547 15696 consensus_queue.cc:1048] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:55.044454 14722 raft_consensus.cc:740] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:55.045063 14442 raft_consensus.cc:2393] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:55.045053 14722 consensus_queue.cc:260] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:55.051997 14722 ts_tablet_manager.cc:1434] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
W20251211 08:57:55.066416 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 97468.4us
I20251211 08:57:55.067189 16013 raft_consensus.cc:493] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4c9cd45be3947d09423ed4508ca2e91)
I20251211 08:57:55.067849 16013 raft_consensus.cc:515] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
W20251211 08:57:54.947360 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 100227us
I20251211 08:57:54.948036 14497 heartbeater.cc:344] Connected to a master server at 127.13.146.62:39015
I20251211 08:57:55.031625 14382 leader_election.cc:304] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:55.071550 15696 raft_consensus.cc:2804] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:55.071949 15696 raft_consensus.cc:697] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:55.073661 15696 consensus_queue.cc:237] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:55.080559 14385 leader_election.cc:343] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:55.084514 16013 leader_election.cc:290] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015), a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977)
I20251211 08:57:55.084465 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" candidate_term: 3 candidate_status { last_received { term: 2 index: 624 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:57:55.097378 14722 tablet_bootstrap.cc:492] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:54.931623 15977 raft_consensus.cc:515] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:55.102126 14447 raft_consensus.cc:2393] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5e624bcfd154cd586ef1dedb9940063 in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
W20251211 08:57:55.110082 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 213846us
I20251211 08:57:55.112104 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:57:54.850558 15940 leader_election.cc:290] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:54.966136 15030 tablet_bootstrap.cc:492] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:55.116820 15030 ts_tablet_manager.cc:1403] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.531s user 0.022s sys 0.037s
I20251211 08:57:54.994597 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 143f20f5c37948068ff2fb0d92f0f34b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:54 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:55.025944 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 715049b6a9214f48a0b573ac4f296424 (DEFAULT_TABLE table=default.Table_bd5960f04a874bee8826f8d460f1388b [id=922194492b5f427aaa346a3ad93b2d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:55.061426 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 1ce1ef2236f54feab5c38bc401c126e8 (DEFAULT_TABLE table=default.Table_afa03065224d44a3a321c94b59f12587 [id=6d0af44f2d4542168eb2af3ead7fe1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:55.064110 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 41ef5bd335a44da39f2549fd55a048f9 (DEFAULT_TABLE table=default.Table_7bce90f465c249f9a5ab2b250fb5610b [id=c6614e8c80da45fda4ee4ea6a7fc7de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:55.093588 15899 raft_consensus.cc:2749] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:55.122524 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce1ef2236f54feab5c38bc401c126e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.123962 15977 leader_election.cc:290] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:55.127471 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ef5bd335a44da39f2549fd55a048f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.133489 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 715049b6a9214f48a0b573ac4f296424. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.133746 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 07099552b546444bab4d66238dd6d146 (DEFAULT_TABLE table=default.Table_3867d5bf46e94dd1990e38dd47037918 [id=ce2647a00ea24733bc2fa5f6cb36ffec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:55.134621 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07099552b546444bab4d66238dd6d146. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.094204 14905 raft_consensus.cc:2272] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:55.095110 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 249960us
I20251211 08:57:55.137044 14714 tablet_bootstrap.cc:654] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:55.137379 14544 leader_election.cc:304] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:55.138939 14905 ts_tablet_manager.cc:1916] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:55.143932 15030 raft_consensus.cc:359] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:55.144616 15030 raft_consensus.cc:385] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:55.144845 15030 raft_consensus.cc:740] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:55.145318 15030 consensus_queue.cc:260] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:57:55.152549 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 52686.7us
I20251211 08:57:55.157644 15969 log.cc:826] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: Log is configured to *not* fsync() on all Append() calls
I20251211 08:57:55.116070 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_39b677204e454017b7cc28e910fa9b30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:55.159814 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39b677204e454017b7cc28e910fa9b30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:55.117174 14010 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_633182cd246844aba2af953c29c5d9e2" } new_table_name: "default.Table_27021f5f53d9422787bd29bc1df43e11" modify_external_catalogs: true
I20251211 08:57:55.119004 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8d772a8a54e2416dacc1df571fe6042c" } modify_external_catalogs: true
I20251211 08:57:55.176959 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 4b9ae463731d4828afc02039e0d161b5 (table default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1]). Expected version 1 got 0
I20251211 08:57:55.180817 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_bdaa13035e5f4ecb80d93694ed51149e [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 1 got 0
I20251211 08:57:55.183096 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:57:55.187577 14012 catalog_manager.cc:5654] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:55.203784 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet b49ea29851d94622b4299e93804a0578 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:54 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:55.200932 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet f5c5b1aebde94dea81cb37a6a6998070 (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]). Expected version 1 got 0
I20251211 08:57:55.205435 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 715049b6a9214f48a0b573ac4f296424 (DEFAULT_TABLE table=default.Table_bd5960f04a874bee8826f8d460f1388b [id=922194492b5f427aaa346a3ad93b2d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:55.206414 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 715049b6a9214f48a0b573ac4f296424. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:55.222146 14502 tablet.cc:2378] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251211 08:57:55.254693 14905 ts_tablet_manager.cc:1929] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:55.122774 14722 tablet_bootstrap.cc:654] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:55.255537 14230 leader_election.cc:343] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: dd14de369e084828afb8a924c46c649b
I20251211 08:57:55.281354 14230 leader_election.cc:304] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:55.273808 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e05628011634149b17d298c54074d00" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:55.284679 14297 raft_consensus.cc:2393] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate c5e624bcfd154cd586ef1dedb9940063 in this term.
I20251211 08:57:55.296777 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6d0fef50524476dbd47fc70e4bda319" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:55.297067 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:55.297442 14298 raft_consensus.cc:3060] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:57:55.299197 15952 raft_consensus.cc:2749] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:55.299409 14263 tablet_service.cc:1505] Processing CreateTablet for tablet 41ef5bd335a44da39f2549fd55a048f9 (DEFAULT_TABLE table=default.Table_7bce90f465c249f9a5ab2b250fb5610b [id=c6614e8c80da45fda4ee4ea6a7fc7de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:55.301069 14263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ef5bd335a44da39f2549fd55a048f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.304888 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 143f20f5c37948068ff2fb0d92f0f34b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:54 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:55.307209 14278 tablet_service.cc:1505] Processing CreateTablet for tablet 1ce1ef2236f54feab5c38bc401c126e8 (DEFAULT_TABLE table=default.Table_afa03065224d44a3a321c94b59f12587 [id=6d0af44f2d4542168eb2af3ead7fe1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:55.308465 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce1ef2236f54feab5c38bc401c126e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.263988 14601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c341b762afd8440bb7198ca134c8dffe" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:55.259997 15970 raft_consensus.cc:493] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:55.265002 16008 tablet.cc:1722] T e354416047f74d02a9455147ca8d6bf3 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:55.256825 15899 raft_consensus.cc:2804] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader election won for term 2
I20251211 08:57:55.316810 15899 raft_consensus.cc:697] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:55.317466 14905 log.cc:1199] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/49015c5d4f2b4d07b547a9df57c992eb
I20251211 08:57:55.269500 15030 ts_tablet_manager.cc:1434] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.152s user 0.004s sys 0.002s
I20251211 08:57:55.318112 14905 ts_tablet_manager.cc:1950] T 49015c5d4f2b4d07b547a9df57c992eb P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:57:55.269966 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 117861us
W20251211 08:57:55.319003 14542 leader_election.cc:343] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: c6d0fef50524476dbd47fc70e4bda319
I20251211 08:57:55.320580 14542 leader_election.cc:304] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:55.321694 14542 leader_election.cc:304] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:55.271445 15902 leader_election.cc:290] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:55.278952 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17ab48721a5c48a7b8a5550be658837d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:55.322645 14608 raft_consensus.cc:2393] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
I20251211 08:57:55.315536 14601 raft_consensus.cc:2393] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:55.323655 14262 tablet_service.cc:1505] Processing CreateTablet for tablet 07099552b546444bab4d66238dd6d146 (DEFAULT_TABLE table=default.Table_3867d5bf46e94dd1990e38dd47037918 [id=ce2647a00ea24733bc2fa5f6cb36ffec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:55.325034 14262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07099552b546444bab4d66238dd6d146. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.280251 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e05628011634149b17d298c54074d00" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:55.317590 15899 consensus_queue.cc:237] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:55.340337 15030 tablet_bootstrap.cc:492] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:55.341696 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 373) took 8547 ms. Trace:
I20251211 08:57:55.342185 14905 rpcz_store.cc:276] 1211 08:57:46.794655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:46.794820 (+ 165us) service_pool.cc:225] Handling call
1211 08:57:54.862561 (+8067741us) ts_tablet_manager.cc:1151] Deleting tablet 49015c5d4f2b4d07b547a9df57c992eb
1211 08:57:54.862606 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.149284 (+286678us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:55.254454 (+105170us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:55.341629 (+ 87175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":230958,"tablet-delete.queue_time_us":7977500}
I20251211 08:57:55.344384 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 379) took 8267 ms. Trace:
I20251211 08:57:55.344743 14905 rpcz_store.cc:276] 1211 08:57:47.076929 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.077052 (+ 123us) service_pool.cc:225] Handling call
1211 08:57:55.343359 (+8266307us) ts_tablet_manager.cc:1151] Deleting tablet 145096302ca24d7e96f2fec45960b18a
1211 08:57:55.344059 (+ 700us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.344337 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8265779}
W20251211 08:57:55.345636 14542 raft_consensus.cc:2686] T 00e5c4bf71b44abca66b0e94995c533e P c5e624bcfd154cd586ef1dedb9940063: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251211 08:57:55.347780 14542 leader_election.cc:304] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:55.348881 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 49015c5d4f2b4d07b547a9df57c992eb (table default.Table_73091d56834e4e78b870257988c87c57 [id=d8f07591554543f2880445cb636da15a]) successfully deleted
I20251211 08:57:55.335171 15970 raft_consensus.cc:515] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:55.354333 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 385) took 7782 ms. Trace:
I20251211 08:57:55.354838 14905 rpcz_store.cc:276] 1211 08:57:47.571608 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.576580 (+ 4972us) service_pool.cc:225] Handling call
1211 08:57:55.353955 (+7777375us) ts_tablet_manager.cc:1151] Deleting tablet 51c3f5eaef784e868e565f81bd51a5e4
1211 08:57:55.354005 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.354284 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7773791}
I20251211 08:57:55.355892 15970 leader_election.cc:290] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:55.356802 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 145096302ca24d7e96f2fec45960b18a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 145096302ca24d7e96f2fec45960b18a already in progress: creating tablet
I20251211 08:57:55.357136 15939 raft_consensus.cc:2804] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:55.357537 15939 raft_consensus.cc:493] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251211 08:57:55.357782 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 51c3f5eaef784e868e565f81bd51a5e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51c3f5eaef784e868e565f81bd51a5e4 already in progress: creating tablet
I20251211 08:57:55.357893 15939 raft_consensus.cc:3060] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:55.356164 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 387) took 7648 ms. Trace:
I20251211 08:57:55.358469 14905 rpcz_store.cc:276] 1211 08:57:47.707839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.710966 (+ 3127us) service_pool.cc:225] Handling call
1211 08:57:55.355753 (+7644787us) ts_tablet_manager.cc:1151] Deleting tablet 8c1e2d6ba1014d00a6df8c6353cff3ab
1211 08:57:55.355811 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.356111 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7644253}
I20251211 08:57:55.363114 15871 raft_consensus.cc:2749] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:57:55.363525 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 8c1e2d6ba1014d00a6df8c6353cff3ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c1e2d6ba1014d00a6df8c6353cff3ab already in progress: creating tablet
I20251211 08:57:55.364327 14905 tablet_replica.cc:333] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:55.365171 14905 raft_consensus.cc:2243] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:55.365720 14905 raft_consensus.cc:2272] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:55.368036 14905 ts_tablet_manager.cc:1916] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:55.373329 14904 ts_tablet_manager.cc:1929] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251211 08:57:55.373868 14904 log.cc:1199] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/e15483f539f24f1ba1b442663d66fd45
I20251211 08:57:55.375125 14904 ts_tablet_manager.cc:1950] T e15483f539f24f1ba1b442663d66fd45 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:55.376142 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 398) took 6124 ms. Trace:
I20251211 08:57:55.376573 14904 rpcz_store.cc:276] 1211 08:57:49.251905 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.252027 (+ 122us) service_pool.cc:225] Handling call
1211 08:57:54.947903 (+5695876us) ts_tablet_manager.cc:1151] Deleting tablet e15483f539f24f1ba1b442663d66fd45
1211 08:57:54.947940 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.115246 (+167306us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:55.333012 (+217766us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:55.376051 (+ 43039us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5695379}
I20251211 08:57:55.377952 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 403) took 5883 ms. Trace:
I20251211 08:57:55.377930 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_49a263a32c9940a5b61d8a430ebb61a7 [id=69dd3ef114034ac7a3d60b8a0fde70b6]) successfully deleted
I20251211 08:57:55.388949 15030 tablet_bootstrap.cc:654] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:55.389618 14008 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_b6a55eca0520454c94dacf9540cb4196 [id=e5ffdeadfa904d85b68d0925da7e7a4c] alter complete (version 2)
I20251211 08:57:55.389878 15970 raft_consensus.cc:493] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:55.392045 15970 raft_consensus.cc:515] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:55.394019 15970 leader_election.cc:290] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:55.378258 14904 rpcz_store.cc:276] 1211 08:57:49.494514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.494629 (+ 115us) service_pool.cc:225] Handling call
1211 08:57:55.377540 (+5882911us) ts_tablet_manager.cc:1151] Deleting tablet c03f976b926a4a6fba8b70e9df9a4160
1211 08:57:55.377600 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.377909 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5882336}
I20251211 08:57:55.408329 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet 143f20f5c37948068ff2fb0d92f0f34b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:54 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:55.402660 15939 raft_consensus.cc:515] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:55.419351 14008 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_cdd8cc77899d40369a8e85dcff2587e2 [id=bf934f83df6142bf8788f1cd2f2142c0] alter complete (version 2)
I20251211 08:57:55.427186 15939 leader_election.cc:290] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:55.427831 14011 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 94) took 2391 ms (client timeout 1000 ms). Trace:
W20251211 08:57:55.440028 14542 leader_election.cc:343] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 45c7d841bd3643a799a8b96e82d6d675
I20251211 08:57:55.444578 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 41ef5bd335a44da39f2549fd55a048f9 (DEFAULT_TABLE table=default.Table_7bce90f465c249f9a5ab2b250fb5610b [id=c6614e8c80da45fda4ee4ea6a7fc7de5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:55.447026 14008 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 57) took 2781 ms (client timeout 1000 ms). Trace:
I20251211 08:57:55.449926 14714 tablet_bootstrap.cc:492] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:55.450379 14714 ts_tablet_manager.cc:1403] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.469s user 0.010s sys 0.004s
I20251211 08:57:55.455717 15030 tablet_bootstrap.cc:492] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:55.459967 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 07099552b546444bab4d66238dd6d146 (DEFAULT_TABLE table=default.Table_3867d5bf46e94dd1990e38dd47037918 [id=ce2647a00ea24733bc2fa5f6cb36ffec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:55.464711 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet f2dc6cb4efc744e6904728ad120df934
W20251211 08:57:55.439951 14007 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 95) took 1242 ms (client timeout 1000 ms). Trace:
I20251211 08:57:55.466766 15030 ts_tablet_manager.cc:1403] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.127s user 0.011s sys 0.003s
W20251211 08:57:55.467095 14007 rpcz_store.cc:269] 1211 08:57:54.197358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:54.280303 (+ 82945us) service_pool.cc:225] Handling call
1211 08:57:54.471986 (+191683us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:55.439889 (+967903us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":18,"spinlock_wait_cycles":385085312}
I20251211 08:57:55.469780 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_8d772a8a54e2416dacc1df571fe6042c" } modify_external_catalogs: true
I20251211 08:57:55.467661 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet f2dc6cb4efc744e6904728ad120df934 with delete_type TABLET_DATA_DELETED (replaced by tablet fb87ed1cb9ac418eaff7676291396ca5) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:55.428263 14011 rpcz_store.cc:269] 1211 08:57:53.036071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.095437 (+ 59366us) service_pool.cc:225] Handling call
1211 08:57:53.153417 (+ 57980us) spinlock_profiling.cc:243] Waited 54.3 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:55.427757 (+2274340us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.137229 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.137558 (+ 329us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.137577 (+ 19us) write_op.cc:435] Acquired schema lock
1211 08:57:54.137587 (+ 10us) tablet.cc:660] Decoding operations
1211 08:57:54.137782 (+ 195us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.137809 (+ 27us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.137827 (+ 18us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:54.137946 (+ 119us) tablet.cc:699] Row locks acquired
1211 08:57:54.137957 (+ 11us) write_op.cc:260] PREPARE: finished
1211 08:57:54.138034 (+ 77us) write_op.cc:270] Start()
1211 08:57:54.138093 (+ 59us) write_op.cc:276] Timestamp: P: 1765443474138156 usec, L: 0
1211 08:57:54.138106 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.138465 (+ 359us) log.cc:844] Serialized 530 byte log entry
1211 08:57:54.246142 (+107677us) op_driver.cc:464] REPLICATION: finished
1211 08:57:54.247010 (+ 868us) write_op.cc:301] APPLY: starting
1211 08:57:54.247056 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:54.247134 (+ 78us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:54.247230 (+ 96us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:54.247380 (+ 150us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:54.247489 (+ 109us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:54.247529 (+ 40us) write_op.cc:312] APPLY: finished
1211 08:57:54.247659 (+ 130us) log.cc:844] Serialized 23 byte log entry
1211 08:57:54.247774 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.247821 (+ 47us) write_op.cc:454] Released schema lock
1211 08:57:54.247846 (+ 25us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:54.248029 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.248049 (+ 20us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":14,"spinlock_wait_cycles":124818688,"child_traces":[["op",{"apply.queue_time_us":740,"apply.run_cpu_time_us":1001,"apply.run_wall_time_us":1000,"num_ops":1,"prepare.queue_time_us":399480,"prepare.run_cpu_time_us":1403,"prepare.run_wall_time_us":1401,"replication_time_us":107936}]]}
I20251211 08:57:55.471678 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_0597df289bfd46ad9628bfdf81aecb3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:55.473201 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0597df289bfd46ad9628bfdf81aecb3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:55.479254 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 605ccab400ed4557bfd7c61cba4448e2 consensus state has no leader
I20251211 08:57:55.479696 14009 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55614 (request call id 535) took 1099 ms. Trace:
I20251211 08:57:55.479986 14009 rpcz_store.cc:276] 1211 08:57:54.380098 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:54.511242 (+131144us) service_pool.cc:225] Handling call
1211 08:57:54.832775 (+321533us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:54.836638 (+ 3863us) catalog_manager.cc:3708] Apply alter schema
1211 08:57:54.837181 (+ 543us) catalog_manager.cc:3740] Acquired catalog manager lock
1211 08:57:54.837363 (+ 182us) catalog_manager.cc:3898] Updating metadata on disk
1211 08:57:55.216998 (+379635us) catalog_manager.cc:3929] Committing alterations to in-memory state
1211 08:57:55.479641 (+262643us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.917242 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.917618 (+ 376us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.917641 (+ 23us) write_op.cc:435] Acquired schema lock
1211 08:57:54.917653 (+ 12us) tablet.cc:660] Decoding operations
1211 08:57:54.917894 (+ 241us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.917928 (+ 34us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.917949 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:54.918102 (+ 153us) tablet.cc:699] Row locks acquired
1211 08:57:54.918114 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:57:54.918215 (+ 101us) write_op.cc:270] Start()
1211 08:57:54.918287 (+ 72us) write_op.cc:276] Timestamp: P: 1765443474918219 usec, L: 0
1211 08:57:54.918300 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.918874 (+ 574us) log.cc:844] Serialized 405 byte log entry
1211 08:57:55.209572 (+290698us) op_driver.cc:464] REPLICATION: finished
1211 08:57:55.211868 (+ 2296us) write_op.cc:301] APPLY: starting
1211 08:57:55.211914 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:55.211991 (+ 77us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:55.212060 (+ 69us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:55.212172 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:55.212280 (+ 108us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:55.212317 (+ 37us) write_op.cc:312] APPLY: finished
1211 08:57:55.212485 (+ 168us) log.cc:844] Serialized 23 byte log entry
1211 08:57:55.212605 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:55.212654 (+ 49us) write_op.cc:454] Released schema lock
1211 08:57:55.212687 (+ 33us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:55.212913 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:55.212942 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2190,"apply.run_cpu_time_us":1009,"apply.run_wall_time_us":1008,"num_ops":1,"prepare.queue_time_us":78867,"prepare.run_cpu_time_us":1841,"prepare.run_wall_time_us":1841,"replication_time_us":291190}]]}
I20251211 08:57:55.484093 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 1ce1ef2236f54feab5c38bc401c126e8 (DEFAULT_TABLE table=default.Table_afa03065224d44a3a321c94b59f12587 [id=6d0af44f2d4542168eb2af3ead7fe1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:55.469219 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_eb245e8feaab450784ce26f23c5575f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:55.491992 14417 tablet_service.cc:1552] Processing DeleteTablet for tablet da52cace2aab4f9dacef18ec5fa56aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:51 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:55.474079 15030 raft_consensus.cc:359] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:55.519644 15030 raft_consensus.cc:385] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:55.519920 14439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1b011cb7c647f79bb16ce1c92bcff5" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:55.520081 15030 raft_consensus.cc:740] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:55.520932 15030 consensus_queue.cc:260] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:57:55.447456 14008 rpcz_store.cc:269] 1211 08:57:52.666025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:52.750326 (+ 84301us) service_pool.cc:225] Handling call
1211 08:57:52.805679 (+ 55353us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:55.446961 (+2641282us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.313476 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.313926 (+ 450us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.313958 (+ 32us) write_op.cc:435] Acquired schema lock
1211 08:57:54.313974 (+ 16us) tablet.cc:660] Decoding operations
1211 08:57:54.314282 (+ 308us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.314325 (+ 43us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.314355 (+ 30us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:54.314515 (+ 160us) tablet.cc:699] Row locks acquired
1211 08:57:54.314531 (+ 16us) write_op.cc:260] PREPARE: finished
1211 08:57:54.314645 (+ 114us) write_op.cc:270] Start()
1211 08:57:54.314732 (+ 87us) write_op.cc:276] Timestamp: P: 1765443474314621 usec, L: 0
1211 08:57:54.314749 (+ 17us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.466825 (+152076us) log.cc:844] Serialized 356 byte log entry
1211 08:57:54.468044 (+ 1219us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbef5a1 00007fc20bc81aa8 00007fc20c0f34d1 00007fc20c0f2ca8 00007fc20c0f7941 00007fc20c0f78f9 00007fc20c0f7889 00007fc20c0f7851 00007fc20c0f6a7d 00000000005aa2b4 00000000005aa0e9
1211 08:57:54.969568 (+501524us) op_driver.cc:464] REPLICATION: finished
1211 08:57:54.970326 (+ 758us) write_op.cc:301] APPLY: starting
1211 08:57:54.970377 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:54.970454 (+ 77us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:54.970471 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:54.970637 (+ 166us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:54.970749 (+ 112us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:54.970811 (+ 62us) write_op.cc:312] APPLY: finished
1211 08:57:54.970981 (+ 170us) log.cc:844] Serialized 23 byte log entry
1211 08:57:54.971109 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.971162 (+ 53us) write_op.cc:454] Released schema lock
1211 08:57:54.971189 (+ 27us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:54.971410 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:54.971429 (+ 19us) write_op.cc:454] Released schema lock
Related trace 'op':
1211 08:57:55.394529 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:55.394919 (+ 390us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:55.394941 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:57:55.394954 (+ 13us) tablet.cc:660] Decoding operations
1211 08:57:55.395191 (+ 237us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:55.395232 (+ 41us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:55.395252 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:55.395384 (+ 132us) tablet.cc:699] Row locks acquired
1211 08:57:55.395397 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:55.395514 (+ 117us) write_op.cc:270] Start()
1211 08:57:55.395601 (+ 87us) write_op.cc:276] Timestamp: P: 1765443475395516 usec, L: 0
1211 08:57:55.395616 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:57:55.396073 (+ 457us) log.cc:844] Serialized 356 byte log entry
1211 08:57:55.410936 (+ 14863us) op_driver.cc:464] REPLICATION: finished
1211 08:57:55.411352 (+ 416us) write_op.cc:301] APPLY: starting
1211 08:57:55.411421 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:55.411533 (+ 112us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:55.411646 (+ 113us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:55.411877 (+ 231us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:55.412065 (+ 188us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:55.412124 (+ 59us) write_op.cc:312] APPLY: finished
1211 08:57:55.412322 (+ 198us) log.cc:844] Serialized 23 byte log entry
1211 08:57:55.412489 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:55.412571 (+ 82us) write_op.cc:454] Released schema lock
1211 08:57:55.412612 (+ 41us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:55.412900 (+ 288us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:55.412931 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":71130,"reported_tablets":13,"spinlock_wait_cycles":87234432,"child_traces":[["op",{"apply.queue_time_us":602,"apply.run_cpu_time_us":1079,"apply.run_wall_time_us":1078,"num_ops":1,"prepare.queue_time_us":18615,"prepare.run_cpu_time_us":4652,"prepare.run_wall_time_us":154687,"replication_time_us":654734,"spinlock_wait_cycles":348892928}],["op",{"apply.queue_time_us":256,"apply.run_cpu_time_us":1496,"apply.run_wall_time_us":1494,"num_ops":1,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":1796,"prepare.run_wall_time_us":1796,"raft.queue_time_us":1083,"raft.run_cpu_time_us":608,"raft.run_wall_time_us":608,"replication_time_us":15213}]]}
I20251211 08:57:55.524646 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88860fa9d7e6414185f0a1cbc78b2571" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
W20251211 08:57:55.489919 14542 leader_election.cc:343] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: c6d0fef50524476dbd47fc70e4bda319
I20251211 08:57:55.525257 14448 raft_consensus.cc:3060] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:55.489069 15899 raft_consensus.cc:493] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:55.526158 15899 raft_consensus.cc:515] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:55.527122 14006 catalog_manager.cc:6724] ReplaceTablet: tablet f2dc6cb4efc744e6904728ad120df934 deleted and replaced by tablet fb87ed1cb9ac418eaff7676291396ca5
I20251211 08:57:55.527500 14006 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55614 (request call id 536) took 1104 ms. Trace:
I20251211 08:57:55.528195 15899 leader_election.cc:290] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:55.489565 15941 ts_tablet_manager.cc:1929] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:55.534484 14714 raft_consensus.cc:359] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:55.535140 15941 log.cc:1199] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/07aef1eea7c04d9cbf35c82e45df7a16
I20251211 08:57:55.535938 15941 ts_tablet_manager.cc:1950] T 07aef1eea7c04d9cbf35c82e45df7a16 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:55.536882 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ef5bd335a44da39f2549fd55a048f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.537232 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 483) took 1720 ms. Trace:
I20251211 08:57:55.537601 15941 rpcz_store.cc:276] 1211 08:57:53.817291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.817381 (+ 90us) service_pool.cc:225] Handling call
1211 08:57:54.480031 (+662650us) ts_tablet_manager.cc:1151] Deleting tablet 07aef1eea7c04d9cbf35c82e45df7a16
1211 08:57:54.480061 (+ 30us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.055629 (+575568us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:55.448362 (+392733us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:55.537186 (+ 88824us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":662158}
I20251211 08:57:55.539019 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce1ef2236f54feab5c38bc401c126e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.543226 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07099552b546444bab4d66238dd6d146. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:55.535121 14714 raft_consensus.cc:385] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:55.543736 14714 raft_consensus.cc:740] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:55.547504 14417 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7775994f684326b6312c802f6b084d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:55.542878 15030 ts_tablet_manager.cc:1434] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.071s user 0.007s sys 0.000s
I20251211 08:57:55.559168 14417 tablet_service.cc:1552] Processing DeleteTablet for tablet 00e5c4bf71b44abca66b0e94995c533e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:55.559777 15030 tablet_bootstrap.cc:492] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:55.556821 14714 consensus_queue.cc:260] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:57:55.565739 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb245e8feaab450784ce26f23c5575f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:55.385874 14298 raft_consensus.cc:2468] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 2.
I20251211 08:57:55.596088 14905 ts_tablet_manager.cc:1929] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:55.596611 14905 log.cc:1199] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/f938359354b545cfbac560b7d7888b9f
I20251211 08:57:55.597227 14439 raft_consensus.cc:2468] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:55.597446 14905 ts_tablet_manager.cc:1950] T f938359354b545cfbac560b7d7888b9f P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:55.605996 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 14fc87cb76024b5b8227dfe9939e1c89 with delete_type TABLET_DATA_DELETED (replaced by tablet df7611ed7bf5459e9fc81d172ed0f57d) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:55.608064 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 00e5c4bf71b44abca66b0e94995c533e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:55.608137 14575 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:55.613435 15030 tablet_bootstrap.cc:654] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:55.614938 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet e9905227c0a844d199362a442e7d630c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:55.618647 14012 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 64) took 802 ms (client timeout 1000 ms). Trace:
I20251211 08:57:55.614529 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:57:55.623162 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 386) took 7947 ms. Trace:
I20251211 08:57:55.631012 14905 rpcz_store.cc:276] 1211 08:57:47.675314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.719452 (+ 44138us) service_pool.cc:225] Handling call
1211 08:57:55.364176 (+7644724us) ts_tablet_manager.cc:1151] Deleting tablet f938359354b545cfbac560b7d7888b9f
1211 08:57:55.364225 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.502489 (+138264us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:55.595853 (+ 93364us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:55.623038 (+ 27185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7644127}
I20251211 08:57:55.632812 14905 tablet_replica.cc:333] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:55.632894 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet f938359354b545cfbac560b7d7888b9f (table default.Table_797be8cbacae4c4e8f75090433cb74dd [id=9dd619e29f44490a9c035c45e103fcf7]) successfully deleted
I20251211 08:57:55.625295 14007 catalog_manager.cc:5654] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:55.633754 14905 raft_consensus.cc:2243] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:55.633942 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_31fcfbed4a2c4e8aa05fb42c261bf1d5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 2 got 0
I20251211 08:57:55.634362 14714 ts_tablet_manager.cc:1434] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.184s user 0.006s sys 0.000s
I20251211 08:57:55.634384 14905 raft_consensus.cc:2272] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:55.635156 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet f5c5b1aebde94dea81cb37a6a6998070 (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]). Expected version 1 got 0
I20251211 08:57:55.636514 14905 ts_tablet_manager.cc:1916] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:55.643142 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_bdaa13035e5f4ecb80d93694ed51149e [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 1 got 0
W20251211 08:57:55.622722 14012 rpcz_store.cc:269] 1211 08:57:54.816283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:54.818237 (+ 1954us) service_pool.cc:225] Handling call
1211 08:57:55.618592 (+800355us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:55.407103 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:55.407525 (+ 422us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:55.407548 (+ 23us) write_op.cc:435] Acquired schema lock
1211 08:57:55.407561 (+ 13us) tablet.cc:660] Decoding operations
1211 08:57:55.407808 (+ 247us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:55.407841 (+ 33us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:55.407865 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:55.408040 (+ 175us) tablet.cc:699] Row locks acquired
1211 08:57:55.408055 (+ 15us) write_op.cc:260] PREPARE: finished
1211 08:57:55.408159 (+ 104us) write_op.cc:270] Start()
1211 08:57:55.430887 (+ 22728us) write_op.cc:276] Timestamp: P: 1765443475408167 usec, L: 0
1211 08:57:55.430910 (+ 23us) op_driver.cc:348] REPLICATION: starting
1211 08:57:55.431450 (+ 540us) log.cc:844] Serialized 530 byte log entry
1211 08:57:55.455185 (+ 23735us) op_driver.cc:464] REPLICATION: finished
1211 08:57:55.455441 (+ 256us) write_op.cc:301] APPLY: starting
1211 08:57:55.455489 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:55.455569 (+ 80us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:55.455652 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:55.455827 (+ 175us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:55.455950 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:55.455987 (+ 37us) write_op.cc:312] APPLY: finished
1211 08:57:55.456150 (+ 163us) log.cc:844] Serialized 23 byte log entry
1211 08:57:55.456299 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:55.456359 (+ 60us) write_op.cc:454] Released schema lock
1211 08:57:55.456389 (+ 30us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:55.456624 (+ 235us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:55.456672 (+ 48us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":136,"apply.run_cpu_time_us":1127,"apply.run_wall_time_us":1125,"num_ops":1,"prepare.queue_time_us":5859,"prepare.run_cpu_time_us":2023,"prepare.run_wall_time_us":25370,"raft.queue_time_us":89,"raft.run_cpu_time_us":573,"raft.run_wall_time_us":572,"replication_time_us":24188}]]}
I20251211 08:57:55.643788 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:57:55.603717 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 485) took 1708 ms. Trace:
I20251211 08:57:55.650401 15941 rpcz_store.cc:276] 1211 08:57:53.855066 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.855166 (+ 100us) service_pool.cc:225] Handling call
1211 08:57:55.538643 (+1683477us) ts_tablet_manager.cc:1151] Deleting tablet 07099552b546444bab4d66238dd6d146
1211 08:57:55.538680 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.563261 (+ 24581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1683013}
I20251211 08:57:55.689463 14448 raft_consensus.cc:2468] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:55.690483 15941 tablet_replica.cc:333] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:55.691402 15941 raft_consensus.cc:2243] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:55.691900 15941 raft_consensus.cc:2272] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:55.693814 15941 ts_tablet_manager.cc:1916] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:55.709513 14007 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_ca17fcc98d9d495e81e4d8186a51deeb [id=7976dac0c05841819cda24f5894629e1] alter complete (version 1)
I20251211 08:57:55.722884 15899 consensus_queue.cc:1048] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:55.718250 14714 tablet_bootstrap.cc:492] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:55.738039 14420 tablet_service.cc:1552] Processing DeleteTablet for tablet f2dc6cb4efc744e6904728ad120df934 with delete_type TABLET_DATA_DELETED (replaced by tablet fb87ed1cb9ac418eaff7676291396ca5) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:55.739239 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 07aef1eea7c04d9cbf35c82e45df7a16 (table default.Table_d40cb8a6b4974e2e9ff982e893a2c552 [id=d6acd0e03f2743349b345927f5ce7a6d]) successfully deleted
W20251211 08:57:55.740056 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 07099552b546444bab4d66238dd6d146 because the tablet was not found. No further retry: Not found: Tablet not found: 07099552b546444bab4d66238dd6d146
W20251211 08:57:55.740394 14080 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.13.146.62:39015 timed out after 0.663s (SENT) [suppressed 1 similar messages]
W20251211 08:57:55.740609 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet ca5eebf2ce3848c79ee2fc21ad2b1486 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca5eebf2ce3848c79ee2fc21ad2b1486 already in progress: creating tablet
W20251211 08:57:55.741098 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet b49ea29851d94622b4299e93804a0578 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b49ea29851d94622b4299e93804a0578 already in progress: creating tablet
W20251211 08:57:55.741627 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 143f20f5c37948068ff2fb0d92f0f34b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 143f20f5c37948068ff2fb0d92f0f34b already in progress: creating tablet
W20251211 08:57:55.742157 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet da52cace2aab4f9dacef18ec5fa56aaf because the tablet was not found. No further retry: Not found: Tablet not found: da52cace2aab4f9dacef18ec5fa56aaf
W20251211 08:57:55.743073 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 5f7775994f684326b6312c802f6b084d because the tablet was not found. No further retry: Not found: Tablet not found: 5f7775994f684326b6312c802f6b084d
I20251211 08:57:55.753242 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 785a5edbda984467b897ee1bbaf1430c
W20251211 08:57:55.758970 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 350604us
W20251211 08:57:55.760334 14542 leader_election.cc:343] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 4ab48f6288494005ad42208e2e97ddf2
I20251211 08:57:55.765053 14542 leader_election.cc:304] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:55.765882 15899 raft_consensus.cc:2804] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:55.766264 15899 raft_consensus.cc:493] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:55.766623 15899 raft_consensus.cc:3060] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:55.773154 14542 leader_election.cc:304] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:55.774396 15939 raft_consensus.cc:2804] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:55.774818 15939 raft_consensus.cc:697] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:55.775521 15939 consensus_queue.cc:237] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:55.793834 14269 tablet_service.cc:1552] Processing DeleteTablet for tablet 49015c5d4f2b4d07b547a9df57c992eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:55.794610 14269 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e05628011634149b17d298c54074d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:55.800292 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 233741us
I20251211 08:57:55.528586 14006 rpcz_store.cc:276] 1211 08:57:54.422789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:54.808983 (+386194us) service_pool.cc:225] Handling call
1211 08:57:55.527447 (+718464us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:54.829595 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:54.829982 (+ 387us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:54.830003 (+ 21us) write_op.cc:435] Acquired schema lock
1211 08:57:54.830014 (+ 11us) tablet.cc:660] Decoding operations
1211 08:57:54.830297 (+ 283us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:54.830329 (+ 32us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:54.830348 (+ 19us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:54.830529 (+ 181us) tablet.cc:699] Row locks acquired
1211 08:57:54.830542 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:54.830652 (+ 110us) write_op.cc:270] Start()
1211 08:57:54.830726 (+ 74us) write_op.cc:276] Timestamp: P: 1765443474830624 usec, L: 0
1211 08:57:54.830738 (+ 12us) op_driver.cc:348] REPLICATION: starting
1211 08:57:54.831239 (+ 501us) log.cc:844] Serialized 625 byte log entry
1211 08:57:55.209356 (+378117us) op_driver.cc:464] REPLICATION: finished
1211 08:57:55.210028 (+ 672us) write_op.cc:301] APPLY: starting
1211 08:57:55.210076 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:55.210186 (+ 110us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:55.210314 (+ 128us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:55.210687 (+ 373us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:55.210920 (+ 233us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:55.210967 (+ 47us) write_op.cc:312] APPLY: finished
1211 08:57:55.211142 (+ 175us) log.cc:844] Serialized 29 byte log entry
1211 08:57:55.211277 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:55.211338 (+ 61us) write_op.cc:454] Released schema lock
1211 08:57:55.211381 (+ 43us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:55.211606 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:55.211627 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":520,"apply.run_cpu_time_us":1568,"apply.run_wall_time_us":1567,"num_ops":2,"prepare.queue_time_us":17256,"prepare.run_cpu_time_us":1837,"prepare.run_wall_time_us":1837,"replication_time_us":378536}]]}
I20251211 08:57:55.802744 14905 ts_tablet_manager.cc:1929] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251211 08:57:55.803114 14905 log.cc:1199] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/1a2dcb9ac62e4aac885d638f4e2b072e
I20251211 08:57:55.803748 14905 ts_tablet_manager.cc:1950] T 1a2dcb9ac62e4aac885d638f4e2b072e P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:55.802254 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_754c328a447a49b8ab7cecad8c5596dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:55.811319 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_754c328a447a49b8ab7cecad8c5596dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:55.812003 15942 consensus_queue.cc:1048] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:55.819278 15969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=11 ignored=0} mutations{seen=7 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251211 08:57:55.820343 15969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: Bootstrap complete.
I20251211 08:57:55.820851 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:55.819875 14904 tablet_replica.cc:333] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:55.827420 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 388) took 7874 ms. Trace:
I20251211 08:57:55.827807 14905 rpcz_store.cc:276] 1211 08:57:47.952930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:47.954067 (+ 1137us) service_pool.cc:225] Handling call
1211 08:57:55.632621 (+7678554us) ts_tablet_manager.cc:1151] Deleting tablet 1a2dcb9ac62e4aac885d638f4e2b072e
1211 08:57:55.632688 (+ 67us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.690257 (+ 57569us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:55.802519 (+112262us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:55.827361 (+ 24842us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7677994}
I20251211 08:57:55.829496 14009 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_479b5ad717dc450689b1fb6ae89169bf" } modify_external_catalogs: true
I20251211 08:57:55.831686 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 393) took 7464 ms. Trace:
I20251211 08:57:55.834985 14905 rpcz_store.cc:276] 1211 08:57:48.367236 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.367344 (+ 108us) service_pool.cc:225] Handling call
1211 08:57:55.831287 (+7463943us) ts_tablet_manager.cc:1151] Deleting tablet f3cf2b4d14ba45d79e52b6fdb0497d58
1211 08:57:55.831345 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.831631 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7374046}
I20251211 08:57:55.837806 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 394) took 7417 ms. Trace:
I20251211 08:57:55.839179 14905 rpcz_store.cc:276] 1211 08:57:48.420840 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.420956 (+ 116us) service_pool.cc:225] Handling call
1211 08:57:55.837372 (+7416416us) ts_tablet_manager.cc:1151] Deleting tablet aadc3c1ecd974ae3a0d491bd209047e6
1211 08:57:55.837438 (+ 66us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:55.837748 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7378369}
W20251211 08:57:55.824792 14722 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.007s sys 0.003s
W20251211 08:57:55.846561 14722 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.563s user 0.007s sys 0.003s
I20251211 08:57:55.830716 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 1a2dcb9ac62e4aac885d638f4e2b072e (table default.Table_0c2ec37766b24f539c7ff1e0aa4ac9c0 [id=1673af33c2f846c7bf092303eff6e8ad]) successfully deleted
I20251211 08:57:55.848174 14009 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_479b5ad717dc450689b1fb6ae89169bf" } modify_external_catalogs: true
I20251211 08:57:55.848459 14275 tablet_service.cc:1552] Processing DeleteTablet for tablet c6d0fef50524476dbd47fc70e4bda319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:48 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:55.848485 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet f3cf2b4d14ba45d79e52b6fdb0497d58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3cf2b4d14ba45d79e52b6fdb0497d58 already in progress: creating tablet
W20251211 08:57:55.849056 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet aadc3c1ecd974ae3a0d491bd209047e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aadc3c1ecd974ae3a0d491bd209047e6 already in progress: creating tablet
W20251211 08:57:55.810415 14010 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eccc727462b447489372da09ea828ac2 consensus state has no leader
I20251211 08:57:55.850510 14274 tablet_service.cc:1552] Processing DeleteTablet for tablet f2dc6cb4efc744e6904728ad120df934 with delete_type TABLET_DATA_DELETED (replaced by tablet fb87ed1cb9ac418eaff7676291396ca5) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:55.850139 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfb3a2b8f8f64c12ab590ffc0be7ace3" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:55.851451 14448 raft_consensus.cc:2468] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:55.851649 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet f785a3cc2e8145eebae7e9c2885b0b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:55.855201 14722 tablet_bootstrap.cc:492] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:55.855780 14722 ts_tablet_manager.cc:1403] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.759s user 0.018s sys 0.004s
W20251211 08:57:55.826995 14230 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.014s sys 0.080s
W20251211 08:57:55.858732 14230 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.536s user 0.014s sys 0.080s
W20251211 08:57:55.859130 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 536767us
W20251211 08:57:55.826942 14297 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.027s sys 0.067s
W20251211 08:57:55.860687 14297 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.561s user 0.028s sys 0.068s
I20251211 08:57:55.861888 14714 tablet_bootstrap.cc:654] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:55.842486 14385 leader_election.cc:304] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:55.864143 14544 leader_election.cc:304] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:55.865880 15950 raft_consensus.cc:493] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:57:55.866386 15950 raft_consensus.cc:515] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:55.868352 15950 leader_election.cc:290] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:55.874527 15942 raft_consensus.cc:2749] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251211 08:57:55.875865 14385 leader_election.cc:304] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:55.876636 15942 raft_consensus.cc:2749] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251211 08:57:55.882818 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e05628011634149b17d298c54074d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:55.888525 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1b011cb7c647f79bb16ce1c92bcff5" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:55.890146 14261 tablet_service.cc:1552] Processing DeleteTablet for tablet 080d1700df4a4df09acda65c6d021d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:55.899785 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:55.900441 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet 785a5edbda984467b897ee1bbaf1430c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:55 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:55.906399 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88860fa9d7e6414185f0a1cbc78b2571" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:55.907246 14298 raft_consensus.cc:3060] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:55.908037 14272 tablet_service.cc:1552] Processing DeleteTablet for tablet da57b72207414512b9744b283bc869fe with delete_type TABLET_DATA_DELETED (replaced by tablet 27cb4228204c4176b1a348960efd5b26) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:55.908519 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet dd14de369e084828afb8a924c46c649b with delete_type TABLET_DATA_DELETED (replaced by tablet ee50f29bfe674611a834d84a7c4e9070) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:55.910609 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 080d1700df4a4df09acda65c6d021d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:55.911751 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet c341b762afd8440bb7198ca134c8dffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:37 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:55.915947 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 785a5edbda984467b897ee1bbaf1430c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:55 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:55.916862 14257 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:46838 (request call id 68) took 1548 ms. Trace:
I20251211 08:57:55.917164 14257 rpcz_store.cc:276] 1211 08:57:54.368709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:54.368827 (+ 118us) service_pool.cc:225] Handling call
1211 08:57:54.645459 (+276632us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54001815d4. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f141cc5a1 00007f0f1425cb78 00007f0f1529d20d 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:57:54.661826 (+ 16367us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b54001711d4. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f141cc5a1 00007f0f1425cb78 00007f0f1529d20d 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:57:54.891577 (+229751us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b5400182d28. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f146046c1 00007f0f146902d2 00007f0f1529d1e0 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:57:55.916693 (+1025116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":842763008}
I20251211 08:57:55.922886 15950 consensus_queue.cc:1048] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:55.953274 14542 leader_election.cc:304] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
W20251211 08:57:55.966478 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d69451d9e49740bb9af84d39d157c2f0 (table default.Table_6ce1c76d3d294fc4ba9a233d7470000e [id=a70a9aab34f347a698c850cb21298eb1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d69451d9e49740bb9af84d39d157c2f0 consensus state has no leader
W20251211 08:57:55.968498 14295 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s user 0.029s sys 0.065s
I20251211 08:57:55.983927 15941 ts_tablet_manager.cc:1929] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:55.984784 15941 log.cc:1199] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/00e5c4bf71b44abca66b0e94995c533e
I20251211 08:57:55.988225 15941 ts_tablet_manager.cc:1950] T 00e5c4bf71b44abca66b0e94995c533e P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:55.992899 14904 raft_consensus.cc:2243] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:55.995811 15941 tablet_replica.cc:333] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:55.999712 14904 raft_consensus.cc:2272] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:56.000491 14722 raft_consensus.cc:359] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:56.001137 14722 raft_consensus.cc:385] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:56.001437 14722 raft_consensus.cc:740] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:56.001547 14904 ts_tablet_manager.cc:1916] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:56.002133 14722 consensus_queue.cc:260] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:56.010922 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a4f1e098ef3e4dc18569e8c2eaac9733" } new_table_name: "default.Table_0db322b6eed54744bc0f600c81c4d1c5" modify_external_catalogs: true
I20251211 08:57:56.011020 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a4dd5045e2db4928b29f19e423ba3188"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:56.007233 15941 raft_consensus.cc:2243] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251211 08:57:56.012529 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4dd5045e2db4928b29f19e423ba3188 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:56.012939 15941 raft_consensus.cc:2272] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:56.016253 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfb3a2b8f8f64c12ab590ffc0be7ace3" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:56.018425 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 65302.2us
I20251211 08:57:56.019423 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 00e5c4bf71b44abca66b0e94995c533e (table default.Table_dcb6d015c5c543689fc117f18bdff260 [id=ff5165bdae864e59a46b71d1ca8d8611]) successfully deleted
I20251211 08:57:56.028035 14298 raft_consensus.cc:2468] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:56.029259 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_3a6c01d8376b45348eb3e5262f0d5d78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:56.030731 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a6c01d8376b45348eb3e5262f0d5d78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:55.905701 14231 leader_election.cc:343] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: da52cace2aab4f9dacef18ec5fa56aaf
W20251211 08:57:56.035063 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 129451us
I20251211 08:57:56.018129 14722 ts_tablet_manager.cc:1434] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.161s user 0.009s sys 0.001s
I20251211 08:57:56.024761 14905 tablet_replica.cc:333] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:56.047658 14497 heartbeater.cc:344] Connected to a master server at 127.13.146.62:39015
I20251211 08:57:56.056167 14722 tablet_bootstrap.cc:492] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:56.087908 14230 leader_election.cc:304] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:56.089116 16037 raft_consensus.cc:3060] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:56.101178 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e71106e05fbc47248f2b5f71482c2231" } modify_external_catalogs: true
I20251211 08:57:56.101475 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b5cff0352c23472291dd9150a225a639"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:56.102352 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_210056e85c404f0082fe3bc88c22c5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:56.102962 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5cff0352c23472291dd9150a225a639 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:56.103704 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_210056e85c404f0082fe3bc88c22c5bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:56.105077 14544 leader_election.cc:343] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Not found: Tablet not found: 4ab48f6288494005ad42208e2e97ddf2
I20251211 08:57:56.105588 14544 leader_election.cc:304] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:56.106106 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_5e9e1c179cbb4953b47e0c0400ee35fc" } new_table_name: "default.Table_5bff583c9d95495aa8ac02605828fa4d" modify_external_catalogs: true
I20251211 08:57:56.101693 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e71106e05fbc47248f2b5f71482c2231" } modify_external_catalogs: true
W20251211 08:57:56.129959 14013 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48575fa3f45349b9887bfc1cdfebc958 consensus state has no leader
I20251211 08:57:56.135754 16037 raft_consensus.cc:2749] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:56.133919 14904 ts_tablet_manager.cc:1929] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:56.149168 14904 log.cc:1199] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/45c7d841bd3643a799a8b96e82d6d675
I20251211 08:57:56.149916 14904 ts_tablet_manager.cc:1950] T 45c7d841bd3643a799a8b96e82d6d675 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:56.183511 14722 tablet_bootstrap.cc:654] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:56.187129 14714 tablet_bootstrap.cc:492] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:56.187492 14714 ts_tablet_manager.cc:1403] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.515s user 0.017s sys 0.005s
W20251211 08:57:56.190411 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 785ccfbc31b74ca6969bcf4020b0520c consensus state has no leader
I20251211 08:57:56.202394 14714 raft_consensus.cc:359] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.208580 14011 master_service.cc:954] ReplaceTablet: received request to replace tablet 476eee1606e74832abf928dbaafc844d from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:56.210198 14722 tablet_bootstrap.cc:492] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:56.210863 14722 ts_tablet_manager.cc:1403] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.155s user 0.014s sys 0.000s
I20251211 08:57:56.214941 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da52cace2aab4f9dacef18ec5fa56aaf" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:56.215453 14606 raft_consensus.cc:2468] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:56.018136 15902 raft_consensus.cc:2804] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader election won for term 2
I20251211 08:57:56.034775 15970 raft_consensus.cc:2804] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:56.221407 14722 raft_consensus.cc:359] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:56.222057 14722 raft_consensus.cc:385] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:56.222313 14722 raft_consensus.cc:740] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:56.222926 14722 consensus_queue.cc:260] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:56.204308 14714 raft_consensus.cc:385] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:56.234650 14714 raft_consensus.cc:740] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:56.033823 15514 consensus_queue.cc:1048] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.195s
I20251211 08:57:56.236205 14714 consensus_queue.cc:260] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.219310 15970 raft_consensus.cc:493] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:56.250082 16044 raft_consensus.cc:2749] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:57:56.262328 15941 ts_tablet_manager.cc:1916] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:56.269493 15950 raft_consensus.cc:493] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:56.274739 15950 raft_consensus.cc:515] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:56.276924 15950 leader_election.cc:290] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:56.278894 14448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5c5b1aebde94dea81cb37a6a6998070" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:56.292332 14714 ts_tablet_manager.cc:1434] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.105s user 0.008s sys 0.000s
I20251211 08:57:56.295536 14722 ts_tablet_manager.cc:1434] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.084s user 0.006s sys 0.000s
I20251211 08:57:56.303547 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 404) took 6807 ms. Trace:
I20251211 08:57:56.303982 14904 rpcz_store.cc:276] 1211 08:57:49.496255 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.497503 (+ 1248us) service_pool.cc:225] Handling call
1211 08:57:55.566832 (+6069329us) ts_tablet_manager.cc:1151] Deleting tablet 45c7d841bd3643a799a8b96e82d6d675
1211 08:57:55.570862 (+ 4030us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.032535 (+461673us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:56.133706 (+101171us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:56.303413 (+169707us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":174482,"tablet-delete.queue_time_us":5903802}
I20251211 08:57:56.305545 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 405) took 6329 ms. Trace:
I20251211 08:57:56.305823 14904 rpcz_store.cc:276] 1211 08:57:49.975849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.975976 (+ 127us) service_pool.cc:225] Handling call
1211 08:57:56.305178 (+6329202us) ts_tablet_manager.cc:1151] Deleting tablet e62ae5b540aa4680b4d18fee29c39e64
1211 08:57:56.305226 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.305499 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6328699}
I20251211 08:57:56.309252 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 45c7d841bd3643a799a8b96e82d6d675 (table default.Table_f1453eb82ce2412c8208000d8ba7f240 [id=f08b1a30da03451ba864cb8f8e6d199b]) successfully deleted
W20251211 08:57:56.309855 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet e62ae5b540aa4680b4d18fee29c39e64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e62ae5b540aa4680b4d18fee29c39e64 already in progress: creating tablet
I20251211 08:57:56.313185 16032 consensus_queue.cc:1048] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:56.257014 15030 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.068s sys 0.089s
I20251211 08:57:56.315587 15996 consensus_queue.cc:1048] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:56.245285 15970 raft_consensus.cc:3060] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:56.306686 14904 tablet_replica.cc:333] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:56.324370 14904 raft_consensus.cc:2243] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:56.326747 14904 raft_consensus.cc:2272] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:56.326920 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2f6e0f1a6bbf4ed1a761612d13f150af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:56.330423 14714 tablet_bootstrap.cc:492] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:56.336048 14722 tablet_bootstrap.cc:492] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:56.274451 15969 raft_consensus.cc:359] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:57:56.350754 14904 ts_tablet_manager.cc:1916] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:56.350958 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f6e0f1a6bbf4ed1a761612d13f150af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:56.360535 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_58973fbd22514d66804dc101b76a7451" } new_table_name: "default.Table_9e007a9469d54d99b9471cb8e653b620" modify_external_catalogs: true
I20251211 08:57:56.322379 15899 raft_consensus.cc:515] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.326509 16052 raft_consensus.cc:493] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:56.367236 16052 raft_consensus.cc:515] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.368798 16052 leader_election.cc:290] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:56.349423 15970 raft_consensus.cc:515] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.370931 15970 leader_election.cc:290] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:56.366653 15899 leader_election.cc:290] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:56.315758 15030 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.677s user 0.068s sys 0.089s
I20251211 08:57:56.373878 15030 tablet_bootstrap.cc:492] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:56.374284 15030 ts_tablet_manager.cc:1403] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.815s user 0.078s sys 0.090s
I20251211 08:57:56.375779 15902 raft_consensus.cc:697] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:56.376731 15902 consensus_queue.cc:237] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:57:56.380393 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 273632us
I20251211 08:57:56.380527 14722 tablet_bootstrap.cc:654] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:56.381000 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7a8acb51a2948f4a018787bfaecb694 consensus state has no leader
I20251211 08:57:56.382431 14230 leader_election.cc:304] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:56.383625 16037 raft_consensus.cc:2804] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:57:56.383930 16037 raft_consensus.cc:493] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:57:56.384289 16037 raft_consensus.cc:3060] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:57:56.386409 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_c1533583a6fd410f96e4da3f3312ba70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:56.338699 15969 raft_consensus.cc:740] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73702d54989f4959acfc4a6fdcede6ca, State: Initialized, Role: FOLLOWER
W20251211 08:57:56.390969 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1533583a6fd410f96e4da3f3312ba70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:56.390852 16037 raft_consensus.cc:515] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:56.394219 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_718ae82148e34936ba20f6d0bebe6169" } new_table_name: "default.Table_ef66c651b9f246749d56bd410712f64e" modify_external_catalogs: true
I20251211 08:57:56.400009 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_13b04a9ecefd46af88c40b23b27dc23b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:56.400890 14231 leader_election.cc:343] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: da52cace2aab4f9dacef18ec5fa56aaf
I20251211 08:57:56.404446 14297 raft_consensus.cc:1275] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Refusing update from remote peer c5e624bcfd154cd586ef1dedb9940063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251211 08:57:56.406303 16026 consensus_queue.cc:1048] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:56.412211 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ae53aa073c864155bd414bdee00d29fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:56.414402 14013 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 677ef16a0610477b8ea05233a040cde4 consensus state has no leader
W20251211 08:57:56.431200 14080 leader_election.cc:336] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 73702d54989f4959acfc4a6fdcede6ca (127.13.146.62:39015): Timed out: RequestConsensusVote RPC to 127.13.146.62:39015 timed out after 0.663s (SENT)
I20251211 08:57:56.431664 14080 leader_election.cc:304] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96e37aef4ef94e33a8a6807c7a9c404f; no voters: 73702d54989f4959acfc4a6fdcede6ca, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:57:56.432788 14714 tablet_bootstrap.cc:654] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:56.394435 15969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
W20251211 08:57:56.442984 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13b04a9ecefd46af88c40b23b27dc23b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:56.445205 16037 leader_election.cc:290] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:56.446697 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae53aa073c864155bd414bdee00d29fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:56.459214 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1b011cb7c647f79bb16ce1c92bcff5" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:56.461241 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3403116517f45c59f87fa5bda0b4071" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:56.462970 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfb3a2b8f8f64c12ab590ffc0be7ace3" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:56.478785 16062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:57:56.481729 16062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: This master's current role is: FOLLOWER
W20251211 08:57:56.486615 14080 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.62:39015, user_credentials={real_user=slave}} blocked reactor thread for 55493us
I20251211 08:57:56.485991 16061 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 96e37aef4ef94e33a8a6807c7a9c404f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251211 08:57:56.492949 15883 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" candidate_term: 3 candidate_status { last_received { term: 2 index: 624 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:56.493026 15884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" candidate_term: 3 candidate_status { last_received { term: 2 index: 624 } } ignore_live_leader: false dest_uuid: "73702d54989f4959acfc4a6fdcede6ca" is_pre_election: true
I20251211 08:57:56.496645 15969 sys_catalog.cc:565] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: configured and running, proceeding with master startup.
I20251211 08:57:56.499122 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da52cace2aab4f9dacef18ec5fa56aaf" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:56.499635 14608 raft_consensus.cc:3060] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 2
I20251211 08:57:56.503085 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5c5b1aebde94dea81cb37a6a6998070" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:56.504724 15883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48334 (request call id 1) took 737 ms (client timeout 663 ms). Trace:
W20251211 08:57:56.505048 15883 rpcz_store.cc:269] 1211 08:57:55.767273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.767390 (+ 117us) service_pool.cc:225] Handling call
1211 08:57:56.504219 (+736829us) spinlock_profiling.cc:243] Waited 9.06 ms on lock 0x7b5800001500. stack: 00007f8abbba5083 00007f8abbba4f80 00007f8abb71a574 0000000000508744 00007f8abba33be1 00007f8abba3e8da 00007f8abba3d229 00007f8ac2c86d90 00007f8ac3cb3647 00007f8ac236b834 00007f8ac236b7c2 00007f8ac236b6f1 00007f8ac236b66c 00007f8ac236a922 00007f8ac098155c 00007f8ac0980996
1211 08:57:56.504646 (+ 427us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20833664}
W20251211 08:57:56.507202 15884 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48334 (request call id 0) took 1295 ms (client timeout 663 ms). Trace:
W20251211 08:57:56.507551 15884 rpcz_store.cc:269] 1211 08:57:55.212174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.212301 (+ 127us) service_pool.cc:225] Handling call
1211 08:57:56.506753 (+1294452us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b5800001500. stack: 00007f8abbba5083 00007f8abbba4f80 00007f8abb71a574 0000000000508744 00007f8abba33be1 00007f8abba3e8da 00007f8abba3d229 00007f8ac2c86d90 00007f8ac3cb3647 00007f8ac236b834 00007f8ac236b7c2 00007f8ac236b6f1 00007f8ac236b66c 00007f8ac236a922 00007f8ac098155c 00007f8ac0980996
1211 08:57:56.507137 (+ 384us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27292416}
I20251211 08:57:56.513463 14608 raft_consensus.cc:2468] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 2.
I20251211 08:57:56.515244 15030 raft_consensus.cc:359] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.515909 15030 raft_consensus.cc:385] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:56.516228 15030 raft_consensus.cc:740] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:56.516971 15030 consensus_queue.cc:260] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.527674 14722 tablet_bootstrap.cc:492] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:56.528083 14722 ts_tablet_manager.cc:1403] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.192s user 0.012s sys 0.004s
I20251211 08:57:56.530129 14722 raft_consensus.cc:359] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.547248 14722 raft_consensus.cc:385] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:56.547555 14722 raft_consensus.cc:740] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
W20251211 08:57:56.548470 14544 leader_election.cc:343] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:56.549465 14904 ts_tablet_manager.cc:1929] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:56.549823 14904 log.cc:1199] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/ccce5a446ac9460c9c9b7de72d875d95
I20251211 08:57:56.550585 14904 ts_tablet_manager.cc:1950] T ccce5a446ac9460c9c9b7de72d875d95 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:57:56.522593 14542 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.006s sys 0.011s
I20251211 08:57:56.549795 14575 tablet_service.cc:1552] Processing DeleteTablet for tablet 785a5edbda984467b897ee1bbaf1430c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:55 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:56.551383 14722 consensus_queue.cc:260] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.552928 14722 ts_tablet_manager.cc:1434] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20251211 08:57:56.554942 14010 master_service.cc:954] ReplaceTablet: received request to replace tablet 07099552b546444bab4d66238dd6d146 from {username='slave'} at 127.0.0.1:55614
W20251211 08:57:56.551314 14542 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s user 0.006s sys 0.011s
W20251211 08:57:56.559026 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 540142us
I20251211 08:57:56.569605 15941 ts_tablet_manager.cc:1929] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:56.570148 15941 log.cc:1199] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/e9905227c0a844d199362a442e7d630c
I20251211 08:57:56.572767 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4c5cbb3bc18748dd81a8641009991654"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:56.579702 14544 leader_election.cc:343] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:56.579180 14714 tablet_bootstrap.cc:492] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
W20251211 08:57:56.582273 14542 consensus_peers.cc:597] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da57b72207414512b9744b283bc869fe. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:56.584136 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_34b8d290de5444d49bed8a5b6ec99f05" } new_table_name: "default.Table_48a06c2ab96d42feb95c8813fb635c80" modify_external_catalogs: true
I20251211 08:57:56.560868 15030 ts_tablet_manager.cc:1434] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.186s user 0.005s sys 0.000s
I20251211 08:57:56.587671 15030 tablet_bootstrap.cc:492] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:56.594077 14905 raft_consensus.cc:2243] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Raft consensus shutting down.
I20251211 08:57:56.607142 14905 pending_rounds.cc:70] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:57:56.607509 14905 pending_rounds.cc:77] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7231256479248314368 op_type: NO_OP noop_request { }
I20251211 08:57:56.607978 14905 raft_consensus.cc:2889] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:56.608372 14905 raft_consensus.cc:2272] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:56.622097 16044 consensus_queue.cc:1048] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:56.626715 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d0cc7be666b34403b1626f0999117efe" } new_table_name: "default.Table_6f2526a7a014488aa1be0b96665abd22" modify_external_catalogs: true
I20251211 08:57:56.633463 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_dbc6fca2f55b4ce9b55985aec36a4a59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:56.635010 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbc6fca2f55b4ce9b55985aec36a4a59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:56.644863 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3403116517f45c59f87fa5bda0b4071" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:56.644865 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1b011cb7c647f79bb16ce1c92bcff5" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:56.646607 14449 raft_consensus.cc:3060] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:56.651913 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c5cbb3bc18748dd81a8641009991654 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:56.652794 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:57:56.654040 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet f5c5b1aebde94dea81cb37a6a6998070 (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]). Expected version 1 got 0
I20251211 08:57:56.654373 14449 raft_consensus.cc:2468] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:56.654975 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (table default.Table_321194b0f71d457e81e14c810650b486 [id=f74e074787144a21ad31d4d89cc509c9]). Expected version 1 got 0
I20251211 08:57:56.655655 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_bdaa13035e5f4ecb80d93694ed51149e [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 1 got 0
I20251211 08:57:56.656137 14012 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:57:56.660107 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 2bf678f74a12464f9361e379e8af0522 (DEFAULT_TABLE table=default.Table_dbdc2eaf4309442888ea49a832da3967 [id=0cd5937f5ed945879b6db098ac98871d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.660933 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf678f74a12464f9361e379e8af0522. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:56.663514 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d6339a0e795499a935b6bf106b5590b consensus state has no leader
I20251211 08:57:56.676396 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_94508c2cd6a741c38c98b9f9f44ee938"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:56.677795 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94508c2cd6a741c38c98b9f9f44ee938 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:56.683085 14905 ts_tablet_manager.cc:1916] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:56.684876 14278 tablet_service.cc:1505] Processing CreateTablet for tablet 2bf678f74a12464f9361e379e8af0522 (DEFAULT_TABLE table=default.Table_dbdc2eaf4309442888ea49a832da3967 [id=0cd5937f5ed945879b6db098ac98871d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.691452 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 1a15b124c6ad49008969737d7cf35f3f (DEFAULT_TABLE table=default.Table_ac8e892bbe604ff89a5b31d2a906eaac [id=1a96483cb56c46f99038958f2e740f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.692468 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a15b124c6ad49008969737d7cf35f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.692950 14004 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (table default.Table_321194b0f71d457e81e14c810650b486 [id=f74e074787144a21ad31d4d89cc509c9]). Expected version 1 got 0
I20251211 08:57:56.693522 14004 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet f5c5b1aebde94dea81cb37a6a6998070 (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]). Expected version 1 got 0
I20251211 08:57:56.694006 14004 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:57:56.696041 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 2bf678f74a12464f9361e379e8af0522 (DEFAULT_TABLE table=default.Table_dbdc2eaf4309442888ea49a832da3967 [id=0cd5937f5ed945879b6db098ac98871d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.697094 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf678f74a12464f9361e379e8af0522. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.707276 14005 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_31fcfbed4a2c4e8aa05fb42c261bf1d5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 2 got 0
I20251211 08:57:56.708604 14005 catalog_manager.cc:5654] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:56.710103 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 412) took 5970 ms. Trace:
I20251211 08:57:56.710207 14005 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:57:56.711288 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf678f74a12464f9361e379e8af0522. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.711442 14722 tablet_bootstrap.cc:492] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:56.716385 14722 tablet_bootstrap.cc:654] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:56.710391 14904 rpcz_store.cc:276] 1211 08:57:50.739223 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:50.739337 (+ 114us) service_pool.cc:225] Handling call
1211 08:57:56.306552 (+5567215us) ts_tablet_manager.cc:1151] Deleting tablet ccce5a446ac9460c9c9b7de72d875d95
1211 08:57:56.306597 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.509293 (+202696us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:56.549231 (+ 39938us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:56.632036 (+ 82805us) spinlock_profiling.cc:243] Waited 78.8 ms on lock 0x7b5800000098. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f0cff7781 00007f0f0d179ac1 00007f0f0d17a0f2 00007f0f15fc7419 00007f0f1521d108 00007f0f152e4462 00007f0f15314f80 00007f0f152fb54d 00007f0f152fb4f9 00007f0f152fb489 00007f0f152fb451
1211 08:57:56.710004 (+ 77968us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":181348224,"tablet-delete.queue_time_us":5566736}
I20251211 08:57:56.719383 14904 tablet_replica.cc:333] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:56.719250 14439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfb3a2b8f8f64c12ab590ffc0be7ace3" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:56.719775 14439 raft_consensus.cc:3060] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:56.720014 14904 raft_consensus.cc:2243] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:56.720867 14904 raft_consensus.cc:2272] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:56.723311 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet ccce5a446ac9460c9c9b7de72d875d95 (table default.Table_afa03065224d44a3a321c94b59f12587 [id=6d0af44f2d4542168eb2af3ead7fe1c4]) successfully deleted
I20251211 08:57:56.723810 14904 ts_tablet_manager.cc:1916] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:56.728936 14005 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
W20251211 08:57:56.729926 14544 consensus_peers.cc:597] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:56.731252 14544 consensus_peers.cc:597] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e15483f539f24f1ba1b442663d66fd45. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:56.732002 14005 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_bdaa13035e5f4ecb80d93694ed51149e [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 1 got 0
I20251211 08:57:56.731990 16044 consensus_queue.cc:1048] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:56.736431 14904 ts_tablet_manager.cc:1929] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:56.736735 14904 log.cc:1199] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/5f7775994f684326b6312c802f6b084d
I20251211 08:57:56.737347 14904 ts_tablet_manager.cc:1950] T 5f7775994f684326b6312c802f6b084d P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:56.741480 15939 raft_consensus.cc:493] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:57:56.741920 15939 raft_consensus.cc:515] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:56.743999 15939 leader_election.cc:290] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:56.744885 14262 tablet_service.cc:1505] Processing CreateTablet for tablet 1a15b124c6ad49008969737d7cf35f3f (DEFAULT_TABLE table=default.Table_ac8e892bbe604ff89a5b31d2a906eaac [id=1a96483cb56c46f99038958f2e740f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.745846 14262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a15b124c6ad49008969737d7cf35f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.746554 14722 tablet_bootstrap.cc:492] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:56.746905 14722 ts_tablet_manager.cc:1403] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.001s
I20251211 08:57:56.749215 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 429) took 5343 ms. Trace:
I20251211 08:57:56.749548 14904 rpcz_store.cc:276] 1211 08:57:51.405537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.410856 (+ 5319us) service_pool.cc:225] Handling call
1211 08:57:56.719246 (+5308390us) ts_tablet_manager.cc:1151] Deleting tablet 5f7775994f684326b6312c802f6b084d
1211 08:57:56.719294 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.729870 (+ 10576us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:56.736246 (+ 6376us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:56.749159 (+ 12913us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5307869}
I20251211 08:57:56.750602 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 5f7775994f684326b6312c802f6b084d (table default.Table_0ff1ebe3d6b144cc94137467d38b30bb [id=e5e62d49ddb749a29892b61cd4615da7]) successfully deleted
I20251211 08:57:56.751735 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 430) took 5300 ms. Trace:
I20251211 08:57:56.582556 14714 ts_tablet_manager.cc:1403] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.252s user 0.013s sys 0.002s
I20251211 08:57:56.752141 14904 rpcz_store.cc:276] 1211 08:57:51.451535 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.452625 (+ 1090us) service_pool.cc:225] Handling call
1211 08:57:56.751282 (+5298657us) ts_tablet_manager.cc:1151] Deleting tablet 63492ba35e26450f9645379a6501d544
1211 08:57:56.751356 (+ 74us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.751672 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5298023}
W20251211 08:57:56.753509 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 63492ba35e26450f9645379a6501d544 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63492ba35e26450f9645379a6501d544 already in progress: creating tablet
I20251211 08:57:56.755661 14722 raft_consensus.cc:359] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.756392 14722 raft_consensus.cc:385] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:56.756657 14722 raft_consensus.cc:740] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:56.759261 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 431) took 5307 ms. Trace:
I20251211 08:57:56.759549 14904 rpcz_store.cc:276] 1211 08:57:51.451806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.451945 (+ 139us) service_pool.cc:225] Handling call
1211 08:57:56.758877 (+5306932us) ts_tablet_manager.cc:1151] Deleting tablet 29869ac852e64489973feb8b0d86ab57
1211 08:57:56.758930 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.759215 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5296021}
W20251211 08:57:56.760680 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 29869ac852e64489973feb8b0d86ab57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29869ac852e64489973feb8b0d86ab57 already in progress: creating tablet
I20251211 08:57:56.763213 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 432) took 5269 ms. Trace:
I20251211 08:57:56.763490 14904 rpcz_store.cc:276] 1211 08:57:51.493971 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.494089 (+ 118us) service_pool.cc:225] Handling call
1211 08:57:56.762866 (+5268777us) ts_tablet_manager.cc:1151] Deleting tablet da124c4afb5a402fb212d0ea95e0b47e
1211 08:57:56.762908 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.763168 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5268209}
W20251211 08:57:56.764390 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet da124c4afb5a402fb212d0ea95e0b47e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da124c4afb5a402fb212d0ea95e0b47e already in progress: creating tablet
W20251211 08:57:56.766074 14544 leader_election.cc:343] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:56.757288 14722 consensus_queue.cc:260] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:57:56.766649 14544 leader_election.cc:343] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:56.767130 14544 leader_election.cc:343] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:56.767886 14722 ts_tablet_manager.cc:1434] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20251211 08:57:56.768633 14722 tablet_bootstrap.cc:492] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:56.771061 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 1a15b124c6ad49008969737d7cf35f3f (DEFAULT_TABLE table=default.Table_ac8e892bbe604ff89a5b31d2a906eaac [id=1a96483cb56c46f99038958f2e740f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.771723 14428 tablet_service.cc:1505] Processing CreateTablet for tablet 28f3a31390aa4c4c92dcce796d21f241 (DEFAULT_TABLE table=default.Table_1610127d3bbc43ae803baa1e8470000f [id=116b25971d944525a48c4d6992d9b8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.772015 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a15b124c6ad49008969737d7cf35f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.772640 14428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f3a31390aa4c4c92dcce796d21f241. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.774160 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 433) took 4027 ms. Trace:
I20251211 08:57:56.774446 14904 rpcz_store.cc:276] 1211 08:57:52.746406 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:52.746531 (+ 125us) service_pool.cc:225] Handling call
1211 08:57:56.773808 (+4027277us) ts_tablet_manager.cc:1151] Deleting tablet ee50f29bfe674611a834d84a7c4e9070
1211 08:57:56.773850 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.774104 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4026723}
I20251211 08:57:56.774813 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 1b2445f66be54d3fbdc3be964b9a9132
W20251211 08:57:56.775431 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet ee50f29bfe674611a834d84a7c4e9070 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee50f29bfe674611a834d84a7c4e9070 already in progress: creating tablet
I20251211 08:57:56.776913 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b2445f66be54d3fbdc3be964b9a9132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:56 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:56.777318 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 436) took 2858 ms. Trace:
I20251211 08:57:56.777582 14904 rpcz_store.cc:276] 1211 08:57:53.918848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.922238 (+ 3390us) service_pool.cc:225] Handling call
1211 08:57:56.777079 (+2854841us) ts_tablet_manager.cc:1151] Deleting tablet 98d14ba15dcd47f8b9e8035b3c7d477c
1211 08:57:56.777118 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.777258 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2854399}
W20251211 08:57:56.778582 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 98d14ba15dcd47f8b9e8035b3c7d477c because the tablet was not found. No further retry: Not found: Tablet not found: 98d14ba15dcd47f8b9e8035b3c7d477c
I20251211 08:57:56.779523 14904 tablet_replica.cc:333] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:56.779472 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b2445f66be54d3fbdc3be964b9a9132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:56 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:56.781293 14904 raft_consensus.cc:2243] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:56.781718 14904 raft_consensus.cc:2272] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:56.783584 14904 ts_tablet_manager.cc:1916] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:56.787338 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b2445f66be54d3fbdc3be964b9a9132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:56 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:56.789537 14722 tablet_bootstrap.cc:654] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:56.793202 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 225438e70aae424a9671aaceb3ee0ed6
I20251211 08:57:56.794413 14009 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55614 (request call id 543) took 1608 ms. Trace:
I20251211 08:57:56.795290 14349 heartbeater.cc:344] Connected to a master server at 127.13.146.62:39015
I20251211 08:57:56.794693 14009 rpcz_store.cc:276] 1211 08:57:55.186492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.800225 (+613733us) service_pool.cc:225] Handling call
1211 08:57:55.848566 (+ 48341us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:55.849708 (+ 1142us) catalog_manager.cc:2815] Modifying in-memory table state
1211 08:57:55.849847 (+ 139us) catalog_manager.cc:2824] Locking tablets
1211 08:57:56.645070 (+795223us) catalog_manager.cc:2838] Removing table and tablets from system table
1211 08:57:56.792821 (+147751us) catalog_manager.cc:2856] Removing table from by-name map
1211 08:57:56.793043 (+ 222us) catalog_manager.cc:2872] Committing in-memory state
1211 08:57:56.793127 (+ 84us) catalog_manager.cc:2876] Aborting table tasks
1211 08:57:56.794364 (+ 1237us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:56.766079 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:56.766458 (+ 379us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:56.766477 (+ 19us) write_op.cc:435] Acquired schema lock
1211 08:57:56.766486 (+ 9us) tablet.cc:660] Decoding operations
1211 08:57:56.767079 (+ 593us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:56.767329 (+ 250us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:56.767349 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:56.767562 (+ 213us) tablet.cc:699] Row locks acquired
1211 08:57:56.767574 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:57:56.767652 (+ 78us) write_op.cc:270] Start()
1211 08:57:56.767716 (+ 64us) write_op.cc:276] Timestamp: P: 1765443476767799 usec, L: 0
1211 08:57:56.767728 (+ 12us) op_driver.cc:348] REPLICATION: starting
1211 08:57:56.768245 (+ 517us) log.cc:844] Serialized 759 byte log entry
1211 08:57:56.774680 (+ 6435us) op_driver.cc:464] REPLICATION: finished
1211 08:57:56.777224 (+ 2544us) write_op.cc:301] APPLY: starting
1211 08:57:56.777265 (+ 41us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:56.777374 (+ 109us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:56.777449 (+ 75us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:56.777672 (+ 223us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:56.777790 (+ 118us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:56.777830 (+ 40us) write_op.cc:312] APPLY: finished
1211 08:57:56.778037 (+ 207us) log.cc:844] Serialized 29 byte log entry
1211 08:57:56.792529 (+ 14492us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:56.792599 (+ 70us) write_op.cc:454] Released schema lock
1211 08:57:56.792632 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2425,"num_ops":2,"prepare.queue_time_us":119486,"raft.queue_time_us":58,"raft.run_cpu_time_us":541,"raft.run_wall_time_us":541,"replication_time_us":6873}]]}
W20251211 08:57:56.796835 14542 leader_election.cc:343] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:56.797192 14542 leader_election.cc:304] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:57:56.798240 14426 tablet_service.cc:1552] Processing DeleteTablet for tablet 225438e70aae424a9671aaceb3ee0ed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:55 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:56.798549 14578 tablet_service.cc:1552] Processing DeleteTablet for tablet 225438e70aae424a9671aaceb3ee0ed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:55 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:56.799110 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet 225438e70aae424a9671aaceb3ee0ed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:55 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:56.799081 15902 raft_consensus.cc:2749] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:56.801391 14278 tablet_service.cc:1505] Processing CreateTablet for tablet 28f3a31390aa4c4c92dcce796d21f241 (DEFAULT_TABLE table=default.Table_1610127d3bbc43ae803baa1e8470000f [id=116b25971d944525a48c4d6992d9b8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.801437 14230 leader_election.cc:304] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:56.802398 16037 raft_consensus.cc:2749] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:56.803402 15030 tablet_bootstrap.cc:654] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:56.805706 16037 consensus_queue.cc:1048] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:56.808748 14426 tablet_service.cc:1505] Processing CreateTablet for tablet 854f344321ea4c85a1b42f9a5632bdaa (DEFAULT_TABLE table=default.Table_5ceb94797f2d4cd2b46dab6921f5a8c9 [id=b9c77a8e21ac430dab6b77152f5e3268]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.809625 14426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 854f344321ea4c85a1b42f9a5632bdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.813138 14262 tablet_service.cc:1505] Processing CreateTablet for tablet 854f344321ea4c85a1b42f9a5632bdaa (DEFAULT_TABLE table=default.Table_5ceb94797f2d4cd2b46dab6921f5a8c9 [id=b9c77a8e21ac430dab6b77152f5e3268]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.816339 15942 consensus_queue.cc:1048] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:56.820017 14904 ts_tablet_manager.cc:1929] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:56.820335 14904 log.cc:1199] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/da52cace2aab4f9dacef18ec5fa56aaf
I20251211 08:57:56.820930 14904 ts_tablet_manager.cc:1950] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:56.820973 14578 tablet_service.cc:1505] Processing CreateTablet for tablet 854f344321ea4c85a1b42f9a5632bdaa (DEFAULT_TABLE table=default.Table_5ceb94797f2d4cd2b46dab6921f5a8c9 [id=b9c77a8e21ac430dab6b77152f5e3268]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.822081 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 854f344321ea4c85a1b42f9a5632bdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.827383 15941 ts_tablet_manager.cc:1950] T e9905227c0a844d199362a442e7d630c P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:56.827792 14420 tablet_service.cc:1505] Processing CreateTablet for tablet 53aac024d68c43bca999eec5bd2def69 (DEFAULT_TABLE table=default.Table_835755f0e12346f38507b3c844df43da [id=48ca7d885ff745a5acfdf1d259736bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.828691 14420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53aac024d68c43bca999eec5bd2def69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.829493 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_485bb3815ac5463185922d1db5a7da52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:56.831041 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_485bb3815ac5463185922d1db5a7da52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:56.754712 14714 raft_consensus.cc:359] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.834554 14714 raft_consensus.cc:385] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:56.834820 14714 raft_consensus.cc:740] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:56.835470 14714 consensus_queue.cc:260] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:56.836583 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f3a31390aa4c4c92dcce796d21f241. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.816411 14230 leader_election.cc:304] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:56.844666 14417 tablet_service.cc:1505] Processing CreateTablet for tablet 03bcfe3a0f0645dab65a92ecc0bcaf19 (DEFAULT_TABLE table=default.Table_4575df1d27154c31927f87fb0346f042 [id=f99b3da70d2f43348aad176dbab84a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.845642 14417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03bcfe3a0f0645dab65a92ecc0bcaf19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.853365 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 5654b34108f546bc9406f8489c6406f4 (DEFAULT_TABLE table=default.Table_e6479b81db9f43c9a99abea4806decd0 [id=8a7cadff8239494d80bb5e461481d5c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.854650 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5654b34108f546bc9406f8489c6406f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.859341 14421 tablet_service.cc:1505] Processing CreateTablet for tablet 4b73a1f96140455eb9ff536eb06973cb (DEFAULT_TABLE table=default.Table_24bf44a5acdf49b3803c8dc014433797 [id=b9427e41fdb24e57813b4bf79410a4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.860320 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b73a1f96140455eb9ff536eb06973cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.867153 14009 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_2d6ce3412a654c938bf9e15598110783" } new_table_name: "default.Table_1abd37bebccc451d8a6e49783ca9dc01" modify_external_catalogs: true
I20251211 08:57:56.868772 14416 tablet_service.cc:1505] Processing CreateTablet for tablet b23795be2dd342ef83093a23d4d9ca4c (DEFAULT_TABLE table=default.Table_41112ce230a642ddb3f81bb1ddbb7916 [id=364bbc1d9e8e46e380f1a607a2874926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.869655 14416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b23795be2dd342ef83093a23d4d9ca4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.839308 14542 leader_election.cc:304] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:56.876726 14423 tablet_service.cc:1505] Processing CreateTablet for tablet bf6bd5e7da044d5fab691bd6ba08bb9f (DEFAULT_TABLE table=default.Table_11ad74b68fec46538d22cec80c57ddd0 [id=e6ca29e163cc42a58f8ce8f0a89a37b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.877983 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf6bd5e7da044d5fab691bd6ba08bb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.887951 14418 tablet_service.cc:1505] Processing CreateTablet for tablet f6e5f7e0ce4e4e9a9f98fdb1d81d27c4 (DEFAULT_TABLE table=default.Table_2bcb516a7ea04fe7be2fd23eb963dd27 [id=ef1bc4f02c544473aef5e896740751d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.889219 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6e5f7e0ce4e4e9a9f98fdb1d81d27c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.892490 14424 tablet_service.cc:1505] Processing CreateTablet for tablet a9d63dc89d7e4fdc8615fe78e0882be7 (DEFAULT_TABLE table=default.Table_6c6a2b03ebbd4c7b9db039b0cfca6997 [id=2f4d9073e99f4b9c8d4dceb6c3de5e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.893671 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9d63dc89d7e4fdc8615fe78e0882be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.903649 14422 tablet_service.cc:1505] Processing CreateTablet for tablet e70de61c3dbb440eb5608abb114d37fd (DEFAULT_TABLE table=default.Table_70f34b083c7d4203859066950bca5c6a [id=5d281b9dfb7d4caeb570ab2c512436a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.904698 14422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e70de61c3dbb440eb5608abb114d37fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.905786 14583 tablet_service.cc:1505] Processing CreateTablet for tablet 53aac024d68c43bca999eec5bd2def69 (DEFAULT_TABLE table=default.Table_835755f0e12346f38507b3c844df43da [id=48ca7d885ff745a5acfdf1d259736bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.907848 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 03bcfe3a0f0645dab65a92ecc0bcaf19 (DEFAULT_TABLE table=default.Table_4575df1d27154c31927f87fb0346f042 [id=f99b3da70d2f43348aad176dbab84a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.909829 14584 tablet_service.cc:1505] Processing CreateTablet for tablet 5654b34108f546bc9406f8489c6406f4 (DEFAULT_TABLE table=default.Table_e6479b81db9f43c9a99abea4806decd0 [id=8a7cadff8239494d80bb5e461481d5c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.911758 14576 tablet_service.cc:1505] Processing CreateTablet for tablet 4b73a1f96140455eb9ff536eb06973cb (DEFAULT_TABLE table=default.Table_24bf44a5acdf49b3803c8dc014433797 [id=b9427e41fdb24e57813b4bf79410a4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.912355 14580 tablet_service.cc:1505] Processing CreateTablet for tablet b23795be2dd342ef83093a23d4d9ca4c (DEFAULT_TABLE table=default.Table_41112ce230a642ddb3f81bb1ddbb7916 [id=364bbc1d9e8e46e380f1a607a2874926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.915149 14011 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 476eee1606e74832abf928dbaafc844d
I20251211 08:57:56.916450 14011 catalog_manager.cc:6724] ReplaceTablet: tablet 476eee1606e74832abf928dbaafc844d deleted and replaced by tablet 62c8dcb36e694cf896bd1283db63b4cc
I20251211 08:57:56.917971 14579 tablet_service.cc:1505] Processing CreateTablet for tablet bf6bd5e7da044d5fab691bd6ba08bb9f (DEFAULT_TABLE table=default.Table_11ad74b68fec46538d22cec80c57ddd0 [id=e6ca29e163cc42a58f8ce8f0a89a37b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.919821 14419 tablet_service.cc:1552] Processing DeleteTablet for tablet 476eee1606e74832abf928dbaafc844d with delete_type TABLET_DATA_DELETED (replaced by tablet 62c8dcb36e694cf896bd1283db63b4cc) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:56.922510 14583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53aac024d68c43bca999eec5bd2def69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.927480 14584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5654b34108f546bc9406f8489c6406f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.927953 14439 raft_consensus.cc:2468] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:56.931448 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03bcfe3a0f0645dab65a92ecc0bcaf19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.936324 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b23795be2dd342ef83093a23d4d9ca4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.939545 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf6bd5e7da044d5fab691bd6ba08bb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.940243 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 80853e3a1ca24fcdb889aee9dd6619f7 (DEFAULT_TABLE table=default.Table_e5b962b03e05467d95482d75eaff0a43 [id=5ea14e5cfb344904a01908d23ddf53c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.941279 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80853e3a1ca24fcdb889aee9dd6619f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.943362 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b73a1f96140455eb9ff536eb06973cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.947371 14429 tablet_service.cc:1505] Processing CreateTablet for tablet c8e25dad5872400a983ff71c2c0573cf (DEFAULT_TABLE table=default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.947721 14419 tablet_service.cc:1505] Processing CreateTablet for tablet ffaed0b6158a43b781a2541c745a1c4b (DEFAULT_TABLE table=default.Table_add1d53c6e304922b7ea478f77bab064 [id=7102693ccdcd46e1935fac502e64efd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.948354 14429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e25dad5872400a983ff71c2c0573cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.951540 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffaed0b6158a43b781a2541c745a1c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:56.953248 14011 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_bdaa13035e5f4ecb80d93694ed51149e" } new_table_name: "default.Table_7ee755c24982464a9d02be456caa647e" modify_external_catalogs: true
I20251211 08:57:56.962795 14415 tablet_service.cc:1505] Processing CreateTablet for tablet 5872a08932b549948ff00504e8962ba4 (DEFAULT_TABLE table=default.Table_7d1e362fc62d433c876de8400a9f30af [id=99cb0a54022d4b30a33cb15688baad63]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:56.972663 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet eccc727462b447489372da09ea828ac2 (table default.Table_27021f5f53d9422787bd29bc1df43e11 [id=04eb7857f3924c79b07b5d2670766644], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet eccc727462b447489372da09ea828ac2 consensus state has no leader
I20251211 08:57:56.987147 14414 tablet_service.cc:1505] Processing CreateTablet for tablet e147eeb943a2452fbc507b5e5a955357 (DEFAULT_TABLE table=default.Table_331eb0e6f45f4df69d7f5752e81e9453 [id=299695e4f25d4a68be283d4e627b4c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.992261 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 28f3a31390aa4c4c92dcce796d21f241 (DEFAULT_TABLE table=default.Table_1610127d3bbc43ae803baa1e8470000f [id=116b25971d944525a48c4d6992d9b8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.992224 14413 tablet_service.cc:1505] Processing CreateTablet for tablet 76f72e4f600b40d6a762103cd70d9e85 (DEFAULT_TABLE table=default.Table_a197b5bcd78844abae3b6d6e25017d53 [id=25cfe399dadf44aaa688c9b4ea664df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:56.993269 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f3a31390aa4c4c92dcce796d21f241. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.003110 14714 ts_tablet_manager.cc:1434] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.250s user 0.006s sys 0.000s
I20251211 08:57:57.007997 15942 raft_consensus.cc:493] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:57.008550 15942 raft_consensus.cc:515] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.009784 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b931f8327b1c4aea804749feb7a41f6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:57.011173 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b931f8327b1c4aea804749feb7a41f6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:57.011314 15942 leader_election.cc:290] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:57.014645 14013 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d36a9570b7c0483a8373043af972d0cc" } modify_external_catalogs: true
I20251211 08:57:57.015101 14013 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_d36a9570b7c0483a8373043af972d0cc" } modify_external_catalogs: true
I20251211 08:57:57.027594 15030 tablet_bootstrap.cc:492] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:57.028323 15030 ts_tablet_manager.cc:1403] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.441s user 0.003s sys 0.007s
I20251211 08:57:57.039521 16050 raft_consensus.cc:493] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:57.040122 16050 raft_consensus.cc:515] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.095363 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_fe31f77461b643beb43b390d2e857446" } modify_external_catalogs: true
I20251211 08:57:57.095335 16044 raft_consensus.cc:2804] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:57.095868 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_fe31f77461b643beb43b390d2e857446" } modify_external_catalogs: true
W20251211 08:57:57.099236 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 260149us
W20251211 08:57:57.119915 13999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 95143.9us
I20251211 08:57:57.129113 14415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5872a08932b549948ff00504e8962ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.131467 14414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e147eeb943a2452fbc507b5e5a955357. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.135293 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f72e4f600b40d6a762103cd70d9e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.136617 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 500) took 1524 ms. Trace:
I20251211 08:57:57.136989 15941 rpcz_store.cc:276] 1211 08:57:55.612455 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.614807 (+ 2352us) service_pool.cc:225] Handling call
1211 08:57:55.995652 (+380845us) ts_tablet_manager.cc:1151] Deleting tablet e9905227c0a844d199362a442e7d630c
1211 08:57:55.995699 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.428435 (+432736us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:56.569255 (+140820us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.136558 (+567303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":380224}
I20251211 08:57:57.138480 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 501) took 1400 ms. Trace:
I20251211 08:57:57.139663 15941 rpcz_store.cc:276] 1211 08:57:55.737795 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.737892 (+ 97us) service_pool.cc:225] Handling call
1211 08:57:57.138134 (+1400242us) ts_tablet_manager.cc:1151] Deleting tablet f2dc6cb4efc744e6904728ad120df934
1211 08:57:57.138174 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.138428 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1399659}
I20251211 08:57:57.141072 14714 tablet_bootstrap.cc:492] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:57.145968 14714 tablet_bootstrap.cc:654] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:57.115276 16026 raft_consensus.cc:493] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:57.155993 14722 tablet_bootstrap.cc:492] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:57.156395 14722 ts_tablet_manager.cc:1403] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.388s user 0.102s sys 0.178s
I20251211 08:57:57.161090 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet e9905227c0a844d199362a442e7d630c (table default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b [id=45f67c67257e428f8ed2f16898ad5fe6]) successfully deleted
W20251211 08:57:57.162412 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f2dc6cb4efc744e6904728ad120df934 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2dc6cb4efc744e6904728ad120df934 already in progress: creating tablet
I20251211 08:57:57.170648 14577 tablet_service.cc:1505] Processing CreateTablet for tablet f6e5f7e0ce4e4e9a9f98fdb1d81d27c4 (DEFAULT_TABLE table=default.Table_2bcb516a7ea04fe7be2fd23eb963dd27 [id=ef1bc4f02c544473aef5e896740751d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.185377 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6e5f7e0ce4e4e9a9f98fdb1d81d27c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.191773 16050 leader_election.cc:290] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:57.193397 14587 tablet_service.cc:1552] Processing DeleteTablet for tablet 476eee1606e74832abf928dbaafc844d with delete_type TABLET_DATA_DELETED (replaced by tablet 62c8dcb36e694cf896bd1283db63b4cc) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:57.193579 14582 tablet_service.cc:1505] Processing CreateTablet for tablet e70de61c3dbb440eb5608abb114d37fd (DEFAULT_TABLE table=default.Table_70f34b083c7d4203859066950bca5c6a [id=5d281b9dfb7d4caeb570ab2c512436a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.194639 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e70de61c3dbb440eb5608abb114d37fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.194900 14587 tablet_service.cc:1505] Processing CreateTablet for tablet ffaed0b6158a43b781a2541c745a1c4b (DEFAULT_TABLE table=default.Table_add1d53c6e304922b7ea478f77bab064 [id=7102693ccdcd46e1935fac502e64efd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.195771 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffaed0b6158a43b781a2541c745a1c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.203938 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 80853e3a1ca24fcdb889aee9dd6619f7 (DEFAULT_TABLE table=default.Table_e5b962b03e05467d95482d75eaff0a43 [id=5ea14e5cfb344904a01908d23ddf53c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.204351 14581 tablet_service.cc:1505] Processing CreateTablet for tablet c8e25dad5872400a983ff71c2c0573cf (DEFAULT_TABLE table=default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.204552 14574 tablet_service.cc:1505] Processing CreateTablet for tablet 5872a08932b549948ff00504e8962ba4 (DEFAULT_TABLE table=default.Table_7d1e362fc62d433c876de8400a9f30af [id=99cb0a54022d4b30a33cb15688baad63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.205874 14586 tablet_service.cc:1505] Processing CreateTablet for tablet a9d63dc89d7e4fdc8615fe78e0882be7 (DEFAULT_TABLE table=default.Table_6c6a2b03ebbd4c7b9db039b0cfca6997 [id=2f4d9073e99f4b9c8d4dceb6c3de5e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.211920 14573 tablet_service.cc:1505] Processing CreateTablet for tablet e147eeb943a2452fbc507b5e5a955357 (DEFAULT_TABLE table=default.Table_331eb0e6f45f4df69d7f5752e81e9453 [id=299695e4f25d4a68be283d4e627b4c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.212930 14573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e147eeb943a2452fbc507b5e5a955357. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.215088 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80853e3a1ca24fcdb889aee9dd6619f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.216115 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e25dad5872400a983ff71c2c0573cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.216272 15941 tablet_replica.cc:333] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:57.216740 14574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5872a08932b549948ff00504e8962ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.217090 15941 raft_consensus.cc:2243] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:57.217515 15941 pending_rounds.cc:70] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:57.217731 15941 pending_rounds.cc:77] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256472585027584 op_type: NO_OP noop_request { }
I20251211 08:57:57.218112 15941 raft_consensus.cc:2889] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:57.218458 15941 raft_consensus.cc:2272] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:57.223721 15941 ts_tablet_manager.cc:1916] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:57.233592 14714 tablet_bootstrap.cc:492] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:57.233952 14714 ts_tablet_manager.cc:1403] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.000s
I20251211 08:57:57.236016 14714 raft_consensus.cc:359] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.236573 14714 raft_consensus.cc:385] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:57.236783 14714 raft_consensus.cc:740] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:57.237299 14714 consensus_queue.cc:260] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.248783 14714 ts_tablet_manager.cc:1434] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20251211 08:57:57.249701 14714 tablet_bootstrap.cc:492] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:57.259037 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9d63dc89d7e4fdc8615fe78e0882be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.265173 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (table default.Table_321194b0f71d457e81e14c810650b486 [id=f74e074787144a21ad31d4d89cc509c9]). Expected version 1 got 0
I20251211 08:57:57.266345 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet f5c5b1aebde94dea81cb37a6a6998070 (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]). Expected version 1 got 0
I20251211 08:57:57.266974 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:57:57.267585 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_31fcfbed4a2c4e8aa05fb42c261bf1d5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 2 got 0
I20251211 08:57:57.268349 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:57:57.269349 14004 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_f66d427103534bd98c3bcc5f0734b392 [id=491e5bc8d1524a7aae54a99a4d39c7f2] alter complete (version 2)
W20251211 08:57:57.272289 14004 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 59) took 1308 ms (client timeout 1000 ms). Trace:
I20251211 08:57:57.274493 14714 tablet_bootstrap.cc:654] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:57.274888 14905 ts_tablet_manager.cc:1929] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251211 08:57:57.275336 14905 log.cc:1199] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/c6d0fef50524476dbd47fc70e4bda319
W20251211 08:57:57.272596 14004 rpcz_store.cc:269] 1211 08:57:55.963817 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.963967 (+ 150us) service_pool.cc:225] Handling call
1211 08:57:57.272217 (+1308250us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:57.111526 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:57.111932 (+ 406us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:57.111955 (+ 23us) write_op.cc:435] Acquired schema lock
1211 08:57:57.111967 (+ 12us) tablet.cc:660] Decoding operations
1211 08:57:57.112199 (+ 232us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:57.112237 (+ 38us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:57.112259 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:57.112387 (+ 128us) tablet.cc:699] Row locks acquired
1211 08:57:57.112400 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:57:57.112498 (+ 98us) write_op.cc:270] Start()
1211 08:57:57.112573 (+ 75us) write_op.cc:276] Timestamp: P: 1765443477112561 usec, L: 0
1211 08:57:57.112588 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:57:57.113114 (+ 526us) log.cc:844] Serialized 356 byte log entry
1211 08:57:57.171295 (+ 58181us) op_driver.cc:464] REPLICATION: finished
1211 08:57:57.171510 (+ 215us) write_op.cc:301] APPLY: starting
1211 08:57:57.171572 (+ 62us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:57.171672 (+ 100us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:57.171774 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:57.171929 (+ 155us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:57.172051 (+ 122us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:57.172090 (+ 39us) write_op.cc:312] APPLY: finished
1211 08:57:57.172247 (+ 157us) log.cc:844] Serialized 23 byte log entry
1211 08:57:57.172375 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:57.172439 (+ 64us) write_op.cc:454] Released schema lock
1211 08:57:57.172467 (+ 28us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:57.183074 (+ 10607us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:57.183097 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":210,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":84,"apply.run_cpu_time_us":1132,"apply.run_wall_time_us":1131,"num_ops":1,"prepare.queue_time_us":382952,"prepare.run_cpu_time_us":1936,"prepare.run_wall_time_us":71528,"raft.queue_time_us":37088,"raft.run_cpu_time_us":614,"raft.run_wall_time_us":614,"replication_time_us":58608}]]}
I20251211 08:57:57.276860 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6856babaa0d64d45902ac1bf4c4f8b0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:57.281150 14905 ts_tablet_manager.cc:1950] T c6d0fef50524476dbd47fc70e4bda319 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:57:57.281507 14012 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 65) took 1473 ms (client timeout 1000 ms). Trace:
W20251211 08:57:57.281858 14012 rpcz_store.cc:269] 1211 08:57:55.807814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.808011 (+ 197us) service_pool.cc:225] Handling call
1211 08:57:57.281443 (+1473432us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9011,"reported_tablets":23,"spinlock_wait_cycles":10496}
W20251211 08:57:57.283732 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6856babaa0d64d45902ac1bf4c4f8b0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:57.284535 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_507f5535a8db4b5ba1c1d46b38a612ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:57.285914 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_507f5535a8db4b5ba1c1d46b38a612ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:57.288137 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet e9905227c0a844d199362a442e7d630c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:57.289000 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:57.290202 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet f785a3cc2e8145eebae7e9c2885b0b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:57.298564 14714 tablet_bootstrap.cc:492] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:57.299052 14714 ts_tablet_manager.cc:1403] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.050s user 0.012s sys 0.000s
I20251211 08:57:57.300144 14722 raft_consensus.cc:359] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.300824 14722 raft_consensus.cc:385] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:57.301350 14722 raft_consensus.cc:740] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:57.301939 14722 consensus_queue.cc:260] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:57:57.305326 14230 raft_consensus.cc:2686] T da52cace2aab4f9dacef18ec5fa56aaf P 31404d5f58e548519239569b150d975b: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251211 08:57:57.305855 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 489753us
I20251211 08:57:57.311995 14263 tablet_service.cc:1505] Processing CreateTablet for tablet 03bcfe3a0f0645dab65a92ecc0bcaf19 (DEFAULT_TABLE table=default.Table_4575df1d27154c31927f87fb0346f042 [id=f99b3da70d2f43348aad176dbab84a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.314915 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 438) took 3379 ms. Trace:
I20251211 08:57:57.315297 14714 raft_consensus.cc:359] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.315222 14904 rpcz_store.cc:276] 1211 08:57:53.935046 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.935132 (+ 86us) service_pool.cc:225] Handling call
1211 08:57:56.779393 (+2844261us) ts_tablet_manager.cc:1151] Deleting tablet da52cace2aab4f9dacef18ec5fa56aaf
1211 08:57:56.779430 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.803628 (+ 24198us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:56.819781 (+ 16153us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.314859 (+495078us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2843276}
I20251211 08:57:57.315495 14260 tablet_service.cc:1505] Processing CreateTablet for tablet 53aac024d68c43bca999eec5bd2def69 (DEFAULT_TABLE table=default.Table_835755f0e12346f38507b3c844df43da [id=48ca7d885ff745a5acfdf1d259736bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.315829 14714 raft_consensus.cc:385] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:57.316058 14714 raft_consensus.cc:740] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:57.316511 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 439) took 3381 ms. Trace:
I20251211 08:57:57.316656 14714 consensus_queue.cc:260] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.157284 16026 raft_consensus.cc:515] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:57.316766 14904 rpcz_store.cc:276] 1211 08:57:53.935313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.937310 (+ 1997us) service_pool.cc:225] Handling call
1211 08:57:57.316206 (+3378896us) ts_tablet_manager.cc:1151] Deleting tablet 07099552b546444bab4d66238dd6d146
1211 08:57:57.316252 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.316469 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3378393}
I20251211 08:57:57.317926 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 443) took 3154 ms. Trace:
I20251211 08:57:57.318187 14904 rpcz_store.cc:276] 1211 08:57:54.163975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:54.164628 (+ 653us) service_pool.cc:225] Handling call
1211 08:57:57.317501 (+3152873us) ts_tablet_manager.cc:1151] Deleting tablet ca5eebf2ce3848c79ee2fc21ad2b1486
1211 08:57:57.317568 (+ 67us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.317881 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3152291}
I20251211 08:57:57.319090 14714 ts_tablet_manager.cc:1434] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20251211 08:57:57.319131 16026 leader_election.cc:290] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:57.320976 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 447) took 2503 ms. Trace:
I20251211 08:57:57.321280 14904 rpcz_store.cc:276] 1211 08:57:54.817991 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:54.818087 (+ 96us) service_pool.cc:225] Handling call
1211 08:57:57.320641 (+2502554us) ts_tablet_manager.cc:1151] Deleting tablet b49ea29851d94622b4299e93804a0578
1211 08:57:57.320695 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.320933 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2502034}
I20251211 08:57:57.322364 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 449) took 2051 ms. Trace:
I20251211 08:57:57.322614 14904 rpcz_store.cc:276] 1211 08:57:55.270796 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.304580 (+ 33784us) service_pool.cc:225] Handling call
1211 08:57:57.322073 (+2017493us) ts_tablet_manager.cc:1151] Deleting tablet 143f20f5c37948068ff2fb0d92f0f34b
1211 08:57:57.322105 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.322309 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2016487}
I20251211 08:57:57.323437 14904 tablet_replica.cc:333] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:57.324096 14904 raft_consensus.cc:2243] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:57.324581 14904 raft_consensus.cc:2272] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:57.326285 14904 ts_tablet_manager.cc:1916] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:57.326342 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 51c3f5eaef784e868e565f81bd51a5e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:57.328404 15950 raft_consensus.cc:493] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:57.302497 14262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 854f344321ea4c85a1b42f9a5632bdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.331713 15950 raft_consensus.cc:515] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.339015 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53aac024d68c43bca999eec5bd2def69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.340068 15950 leader_election.cc:290] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:57.350831 14231 leader_election.cc:343] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 00e5c4bf71b44abca66b0e94995c533e
I20251211 08:57:57.350313 14263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03bcfe3a0f0645dab65a92ecc0bcaf19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.352262 14722 ts_tablet_manager.cc:1434] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.196s user 0.001s sys 0.012s
I20251211 08:57:57.353106 14722 tablet_bootstrap.cc:492] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:57.357352 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfb3a2b8f8f64c12ab590ffc0be7ace3" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:57.358079 14441 raft_consensus.cc:2393] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 1: Already voted for candidate c5e624bcfd154cd586ef1dedb9940063 in this term.
I20251211 08:57:57.363909 14714 tablet_bootstrap.cc:492] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:57.378795 14009 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58d5ded96b9149f9ab0d9ea70e376af1 consensus state has no leader
I20251211 08:57:57.382593 14722 tablet_bootstrap.cc:654] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:57.388185 14714 tablet_bootstrap.cc:654] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:57.391405 15941 ts_tablet_manager.cc:1929] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:57.391794 15941 log.cc:1199] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/8e05628011634149b17d298c54074d00
I20251211 08:57:57.392459 15941 ts_tablet_manager.cc:1950] T 8e05628011634149b17d298c54074d00 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:57.393159 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 502) took 1511 ms. Trace:
I20251211 08:57:57.393465 15941 rpcz_store.cc:276] 1211 08:57:55.881892 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.882675 (+ 783us) service_pool.cc:225] Handling call
1211 08:57:57.140565 (+1257890us) ts_tablet_manager.cc:1151] Deleting tablet 8e05628011634149b17d298c54074d00
1211 08:57:57.216146 (+ 75581us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.298440 (+ 82294us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.391203 (+ 92763us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.393115 (+ 1912us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4992,"tablet-delete.queue_time_us":1257296}
I20251211 08:57:57.401392 14714 tablet_bootstrap.cc:492] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:57.401784 14714 ts_tablet_manager.cc:1403] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.006s
I20251211 08:57:57.403730 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 8e05628011634149b17d298c54074d00 (table default.Table_885d3fb0ab5f4d0a82972d94a34c2a88 [id=18d1df6385a24d26bfb44309aa58eaa9]) successfully deleted
I20251211 08:57:57.404213 14714 raft_consensus.cc:359] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.404495 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 503) took 1505 ms. Trace:
I20251211 08:57:57.404769 14714 raft_consensus.cc:385] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:57.405092 14714 raft_consensus.cc:740] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:57.405753 14714 consensus_queue.cc:260] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.408608 14714 ts_tablet_manager.cc:1434] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20251211 08:57:57.409497 14714 tablet_bootstrap.cc:492] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:57.412835 15942 raft_consensus.cc:493] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:57.413206 15942 raft_consensus.cc:515] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.415448 15942 leader_election.cc:290] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:57.418023 14904 ts_tablet_manager.cc:1929] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:57.428462 14904 log.cc:1199] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/49015c5d4f2b4d07b547a9df57c992eb
W20251211 08:57:57.429713 14011 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88860fa9d7e6414185f0a1cbc78b2571 consensus state has no leader
I20251211 08:57:57.429878 14904 ts_tablet_manager.cc:1950] T 49015c5d4f2b4d07b547a9df57c992eb P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:57.437842 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet f3403116517f45c59f87fa5bda0b4071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:57.443199 15030 raft_consensus.cc:359] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.443763 15030 raft_consensus.cc:385] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:57.445173 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 395) took 8999 ms. Trace:
I20251211 08:57:57.445469 14905 rpcz_store.cc:276] 1211 08:57:48.445327 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.445422 (+ 95us) service_pool.cc:225] Handling call
1211 08:57:55.923821 (+7478399us) ts_tablet_manager.cc:1151] Deleting tablet c6d0fef50524476dbd47fc70e4bda319
1211 08:57:56.024645 (+100824us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:56.597021 (+572376us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b54000f0d54. stack: 00007f0c13f80083 00007f0c13f7ff80 00007f0c13af5574 0000000000508744 00000000005086d9 00007f0c1afc75a1 00007f0c1b065fe2 00007f0c1b4882ec 00007f0c1c0df31a 00007f0c1c10ff80 00007f0c1c0f654d 00007f0c1c0f64f9 00007f0c1c0f6489 00007f0c1c0f6451 00007f0c1c0f567d 00000000005aa2b4
1211 08:57:56.693403 (+ 96382us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.255385 (+561982us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.445122 (+189737us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9589,"spinlock_wait_cycles":867059840,"tablet-delete.queue_time_us":7464633}
I20251211 08:57:57.444015 15030 raft_consensus.cc:740] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:57.257970 16044 raft_consensus.cc:697] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:57.447256 15030 consensus_queue.cc:260] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.451545 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet c6d0fef50524476dbd47fc70e4bda319 (table default.Table_7919ce70cfc5400bb4f2515164c6b7a1 [id=3a336d5ad7004619951e901ad9bf39bf]) successfully deleted
I20251211 08:57:57.455981 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 396) took 9010 ms. Trace:
I20251211 08:57:57.456344 14905 rpcz_store.cc:276] 1211 08:57:48.446024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:48.446123 (+ 99us) service_pool.cc:225] Handling call
1211 08:57:57.446435 (+9000312us) ts_tablet_manager.cc:1151] Deleting tablet 9b5a4b6b1b28406c82e2acd9356b60d4
1211 08:57:57.450567 (+ 4132us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.455915 (+ 5348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8987123}
W20251211 08:57:57.459275 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 359340us
I20251211 08:57:57.459837 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 457) took 1666 ms. Trace:
I20251211 08:57:57.460227 14904 rpcz_store.cc:276] 1211 08:57:55.793417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.793580 (+ 163us) service_pool.cc:225] Handling call
1211 08:57:57.323324 (+1529744us) ts_tablet_manager.cc:1151] Deleting tablet 49015c5d4f2b4d07b547a9df57c992eb
1211 08:57:57.323355 (+ 31us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.370100 (+ 46745us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.417793 (+ 47693us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.459769 (+ 41976us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1529063}
I20251211 08:57:57.461395 14904 tablet_replica.cc:333] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b: stopping tablet replica
W20251211 08:57:57.461920 14542 consensus_peers.cc:597] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e15483f539f24f1ba1b442663d66fd45. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:57.472469 14013 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 15595c5a432047de873c2647b78838d6
W20251211 08:57:57.483722 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 9b5a4b6b1b28406c82e2acd9356b60d4 because the tablet was not found. No further retry: Not found: Tablet not found: 9b5a4b6b1b28406c82e2acd9356b60d4
W20251211 08:57:57.434628 14005 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 96) took 1211 ms (client timeout 1000 ms). Trace:
I20251211 08:57:57.490878 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 15595c5a432047de873c2647b78838d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:57.490093 14005 rpcz_store.cc:269] 1211 08:57:56.222958 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:56.223088 (+ 130us) service_pool.cc:225] Handling call
1211 08:57:57.434573 (+1211485us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:57.183355 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:57.183899 (+ 544us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:57.183933 (+ 34us) write_op.cc:435] Acquired schema lock
1211 08:57:57.183953 (+ 20us) tablet.cc:660] Decoding operations
1211 08:57:57.184273 (+ 320us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:57.184328 (+ 55us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:57.184351 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:57.184531 (+ 180us) tablet.cc:699] Row locks acquired
1211 08:57:57.184552 (+ 21us) write_op.cc:260] PREPARE: finished
1211 08:57:57.184687 (+ 135us) write_op.cc:270] Start()
1211 08:57:57.184794 (+ 107us) write_op.cc:276] Timestamp: P: 1765443477184767 usec, L: 0
1211 08:57:57.184827 (+ 33us) op_driver.cc:348] REPLICATION: starting
1211 08:57:57.223870 (+ 39043us) log.cc:844] Serialized 530 byte log entry
1211 08:57:57.230077 (+ 6207us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbef5a1 00007fc20bc81aa8 00007fc20c0f34d1 00007fc20c0f2ca8 00007fc20c0f7941 00007fc20c0f78f9 00007fc20c0f7889 00007fc20c0f7851 00007fc20c0f6a7d 00000000005aa2b4 00000000005aa0e9
1211 08:57:57.249888 (+ 19811us) op_driver.cc:464] REPLICATION: finished
1211 08:57:57.250093 (+ 205us) write_op.cc:301] APPLY: starting
1211 08:57:57.250145 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:57.250222 (+ 77us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:57.250240 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:57.250402 (+ 162us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:57.250521 (+ 119us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:57.250560 (+ 39us) write_op.cc:312] APPLY: finished
1211 08:57:57.250722 (+ 162us) log.cc:844] Serialized 23 byte log entry
1211 08:57:57.250880 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:57.250936 (+ 56us) write_op.cc:454] Released schema lock
1211 08:57:57.250970 (+ 34us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:57.294957 (+ 43987us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:57.294994 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":28070,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":74,"apply.run_cpu_time_us":1070,"apply.run_wall_time_us":1070,"num_ops":1,"prepare.queue_time_us":449083,"prepare.run_cpu_time_us":4171,"prepare.run_wall_time_us":111572,"raft.queue_time_us":12388,"raft.run_cpu_time_us":664,"raft.run_wall_time_us":665,"replication_time_us":64967,"spinlock_wait_cycles":88468864}]]}
I20251211 08:57:57.404780 15941 rpcz_store.cc:276] 1211 08:57:55.899418 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.899642 (+ 224us) service_pool.cc:225] Handling call
1211 08:57:57.404258 (+1504616us) ts_tablet_manager.cc:1151] Deleting tablet 07aef1eea7c04d9cbf35c82e45df7a16
1211 08:57:57.404301 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.404448 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1503251}
W20251211 08:57:57.509685 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 07aef1eea7c04d9cbf35c82e45df7a16 because the tablet was not found. No further retry: Not found: Tablet not found: 07aef1eea7c04d9cbf35c82e45df7a16
I20251211 08:57:57.510150 15941 tablet_replica.cc:333] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:57.510999 15941 raft_consensus.cc:2243] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:57.511502 15941 raft_consensus.cc:2272] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:57.516970 15941 ts_tablet_manager.cc:1916] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:57.522565 14714 tablet_bootstrap.cc:654] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:57.535832 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 2c0f8d7dd90f4beaae674fe827d37e0b
I20251211 08:57:57.542040 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c0f8d7dd90f4beaae674fe827d37e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:57.552553 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_38faf33a40be4a6fa97f912efdc62b1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:57.554093 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38faf33a40be4a6fa97f912efdc62b1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:57.587599 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e2bf2dd74d94446aa1dd883c2a18ad68" } new_table_name: "default.Table_8d7c119ecb1e4470b2015b8a77780805" modify_external_catalogs: true
I20251211 08:57:57.507532 16096 raft_consensus.cc:493] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:57:57.508093 14722 tablet_bootstrap.cc:492] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:57.508955 14292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07aef1eea7c04d9cbf35c82e45df7a16" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:57.524720 14290 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "033c2c016f8f4c01adc8bdbf2885e394" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:57.615338 15942 raft_consensus.cc:493] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:57.615818 15942 raft_consensus.cc:515] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.617642 15942 leader_election.cc:290] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:57.463356 14576 tablet_service.cc:1505] Processing CreateTablet for tablet 76f72e4f600b40d6a762103cd70d9e85 (DEFAULT_TABLE table=default.Table_a197b5bcd78844abae3b6d6e25017d53 [id=25cfe399dadf44aaa688c9b4ea664df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.619410 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f72e4f600b40d6a762103cd70d9e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.466166 14905 tablet_replica.cc:333] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:57.455399 16044 consensus_queue.cc:237] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.482975 14581 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a2dcb9ac62e4aac885d638f4e2b072e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:57.483999 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet f3403116517f45c59f87fa5bda0b4071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:57.488605 14575 tablet_service.cc:1552] Processing DeleteTablet for tablet 15595c5a432047de873c2647b78838d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:57.576053 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c0f8d7dd90f4beaae674fe827d37e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:57.631783 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 111adfd1ac1a42bdafa9416594e7e932 consensus state has no leader
I20251211 08:57:57.635993 15942 raft_consensus.cc:493] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251211 08:57:57.602941 14579 consensus_meta.cc:287] T bf6bd5e7da044d5fab691bd6ba08bb9f P c5e624bcfd154cd586ef1dedb9940063: Time spent flushing consensus metadata: real 0.608s user 0.003s sys 0.004s
I20251211 08:57:57.603614 15030 ts_tablet_manager.cc:1434] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.575s user 0.005s sys 0.000s
W20251211 08:57:57.637405 15030 ts_tablet_manager.cc:1467] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: Tablet startup took 608ms
I20251211 08:57:57.638276 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_05c727e9a723434ba4239e838c6afc9e" } new_table_name: "default.Table_5b713a62983c42f4927bf2e96686ced0" modify_external_catalogs: true
I20251211 08:57:57.636415 15942 raft_consensus.cc:515] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:57:57.637774 15030 ts_tablet_manager.cc:1469] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: Trace:
1211 08:57:39.386802 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:39.390836 (+ 4034us) service_pool.cc:225] Handling call
1211 08:57:39.392047 (+ 1211us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1211 08:57:39.392119 (+ 72us) ts_tablet_manager.cc:682] Creating new metadata...
1211 08:57:39.448065 (+ 55946us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:39.474516 (+ 26451us) tablet_replica.cc:206] Creating consensus instance
1211 08:57:39.497218 (+ 22702us) inbound_call.cc:173] Queueing success response
1211 08:57:56.587405 (+17090187us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1211 08:57:57.028692 (+441287us) ts_tablet_manager.cc:1435] Starting tablet replica
1211 08:57:57.029286 (+ 594us) tablet_replica.cc:261] Starting instrumentation
1211 08:57:57.184560 (+155274us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":253798,"raft.queue_time_us":16025,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":138,"tablet-open.queue_time_us":17112607}
I20251211 08:57:57.640638 15942 leader_election.cc:290] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:57.641438 15030 tablet_bootstrap.cc:492] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:57.645010 14905 raft_consensus.cc:2243] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:57.645464 14905 raft_consensus.cc:2272] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:57.646333 15030 tablet_bootstrap.cc:654] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:57.647316 14905 ts_tablet_manager.cc:1916] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:57.659160 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62c8dcb36e694cf896bd1283db63b4cc has no consensus state
W20251211 08:57:57.660485 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 198739us
I20251211 08:57:57.525719 14278 tablet_service.cc:1505] Processing CreateTablet for tablet 5654b34108f546bc9406f8489c6406f4 (DEFAULT_TABLE table=default.Table_e6479b81db9f43c9a99abea4806decd0 [id=8a7cadff8239494d80bb5e461481d5c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.526727 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 4b73a1f96140455eb9ff536eb06973cb (DEFAULT_TABLE table=default.Table_24bf44a5acdf49b3803c8dc014433797 [id=b9427e41fdb24e57813b4bf79410a4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.528080 14272 tablet_service.cc:1505] Processing CreateTablet for tablet b23795be2dd342ef83093a23d4d9ca4c (DEFAULT_TABLE table=default.Table_41112ce230a642ddb3f81bb1ddbb7916 [id=364bbc1d9e8e46e380f1a607a2874926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.530066 14267 tablet_service.cc:1505] Processing CreateTablet for tablet bf6bd5e7da044d5fab691bd6ba08bb9f (DEFAULT_TABLE table=default.Table_11ad74b68fec46538d22cec80c57ddd0 [id=e6ca29e163cc42a58f8ce8f0a89a37b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.535619 14274 tablet_service.cc:1505] Processing CreateTablet for tablet a9d63dc89d7e4fdc8615fe78e0882be7 (DEFAULT_TABLE table=default.Table_6c6a2b03ebbd4c7b9db039b0cfca6997 [id=2f4d9073e99f4b9c8d4dceb6c3de5e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.541508 14261 tablet_service.cc:1505] Processing CreateTablet for tablet f6e5f7e0ce4e4e9a9f98fdb1d81d27c4 (DEFAULT_TABLE table=default.Table_2bcb516a7ea04fe7be2fd23eb963dd27 [id=ef1bc4f02c544473aef5e896740751d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.543288 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51c3f5eaef784e868e565f81bd51a5e4" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:57.546247 14275 tablet_service.cc:1505] Processing CreateTablet for tablet e70de61c3dbb440eb5608abb114d37fd (DEFAULT_TABLE table=default.Table_70f34b083c7d4203859066950bca5c6a [id=5d281b9dfb7d4caeb570ab2c512436a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.548676 14271 tablet_service.cc:1552] Processing DeleteTablet for tablet 476eee1606e74832abf928dbaafc844d with delete_type TABLET_DATA_DELETED (replaced by tablet 62c8dcb36e694cf896bd1283db63b4cc) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:57.559428 14269 tablet_service.cc:1505] Processing CreateTablet for tablet ffaed0b6158a43b781a2541c745a1c4b (DEFAULT_TABLE table=default.Table_add1d53c6e304922b7ea478f77bab064 [id=7102693ccdcd46e1935fac502e64efd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.562031 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 80853e3a1ca24fcdb889aee9dd6619f7 (DEFAULT_TABLE table=default.Table_e5b962b03e05467d95482d75eaff0a43 [id=5ea14e5cfb344904a01908d23ddf53c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.563258 14279 tablet_service.cc:1505] Processing CreateTablet for tablet c8e25dad5872400a983ff71c2c0573cf (DEFAULT_TABLE table=default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.563875 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 76f72e4f600b40d6a762103cd70d9e85 (DEFAULT_TABLE table=default.Table_a197b5bcd78844abae3b6d6e25017d53 [id=25cfe399dadf44aaa688c9b4ea664df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.564342 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 5872a08932b549948ff00504e8962ba4 (DEFAULT_TABLE table=default.Table_7d1e362fc62d433c876de8400a9f30af [id=99cb0a54022d4b30a33cb15688baad63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.569825 14259 tablet_service.cc:1552] Processing DeleteTablet for tablet 45c7d841bd3643a799a8b96e82d6d675 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:57.570431 14265 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7775994f684326b6312c802f6b084d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:57.571031 14268 tablet_service.cc:1505] Processing CreateTablet for tablet e147eeb943a2452fbc507b5e5a955357 (DEFAULT_TABLE table=default.Table_331eb0e6f45f4df69d7f5752e81e9453 [id=299695e4f25d4a68be283d4e627b4c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:57.579854 16093 raft_consensus.cc:493] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:57.615823 14722 ts_tablet_manager.cc:1403] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.263s user 0.007s sys 0.002s
I20251211 08:57:57.668213 14714 tablet_bootstrap.cc:492] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:57.668958 14714 ts_tablet_manager.cc:1403] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.260s user 0.006s sys 0.003s
I20251211 08:57:57.671137 14714 raft_consensus.cc:359] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.671749 14714 raft_consensus.cc:385] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:57.671995 14714 raft_consensus.cc:740] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:57.672698 14714 consensus_queue.cc:260] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.677161 14258 tablet_service.cc:1552] Processing DeleteTablet for tablet e9905227c0a844d199362a442e7d630c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:57.681115 14714 ts_tablet_manager.cc:1434] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20251211 08:57:57.681495 14722 raft_consensus.cc:359] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.682332 14714 tablet_bootstrap.cc:492] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:57.682310 14722 raft_consensus.cc:385] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:57.664954 16096 raft_consensus.cc:515] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:57.685101 16096 leader_election.cc:290] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:57.687515 14714 tablet_bootstrap.cc:654] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:57.687932 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a2712ccf332e4a14a29f57badb7591cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:57.689392 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2712ccf332e4a14a29f57badb7591cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:57.690183 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_dbc6fca2f55b4ce9b55985aec36a4a59" } modify_external_catalogs: true
I20251211 08:57:57.690629 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_dbc6fca2f55b4ce9b55985aec36a4a59" } modify_external_catalogs: true
I20251211 08:57:57.672763 16093 raft_consensus.cc:515] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.693503 16093 leader_election.cc:290] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:57.696100 14292 raft_consensus.cc:2468] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
W20251211 08:57:57.698613 14231 leader_election.cc:343] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 07aef1eea7c04d9cbf35c82e45df7a16
I20251211 08:57:57.700174 14439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3403116517f45c59f87fa5bda0b4071" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:57.700706 14439 raft_consensus.cc:2468] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:57:57.704938 15030 tablet_bootstrap.cc:492] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:57.705292 15030 ts_tablet_manager.cc:1403] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.001s
I20251211 08:57:57.712265 14231 leader_election.cc:304] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:57.717754 15941 ts_tablet_manager.cc:1929] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:57.718164 15941 log.cc:1199] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/080d1700df4a4df09acda65c6d021d96
I20251211 08:57:57.718928 15941 ts_tablet_manager.cc:1950] T 080d1700df4a4df09acda65c6d021d96 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:57.720098 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5654b34108f546bc9406f8489c6406f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.720276 16093 raft_consensus.cc:2804] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:57.720688 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 504) took 1815 ms. Trace:
I20251211 08:57:57.720602 16093 raft_consensus.cc:493] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:57.720994 16093 raft_consensus.cc:3060] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:57.720947 15941 rpcz_store.cc:276] 1211 08:57:55.904822 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.910453 (+ 5631us) service_pool.cc:225] Handling call
1211 08:57:57.509998 (+1599545us) ts_tablet_manager.cc:1151] Deleting tablet 080d1700df4a4df09acda65c6d021d96
1211 08:57:57.510042 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.662194 (+152152us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.717511 (+ 55317us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.720640 (+ 3129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598879}
I20251211 08:57:57.721971 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 080d1700df4a4df09acda65c6d021d96 (table default.Table_26f4e336df6947149cb311c15bee56c1 [id=e647fb169f024aebb1f942fff0779130]) successfully deleted
I20251211 08:57:57.723307 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b23795be2dd342ef83093a23d4d9ca4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.727092 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf6bd5e7da044d5fab691bd6ba08bb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.727581 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e25dad5872400a983ff71c2c0573cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.728188 14261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6e5f7e0ce4e4e9a9f98fdb1d81d27c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.682621 14722 raft_consensus.cc:740] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:57.742658 14012 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_27021f5f53d9422787bd29bc1df43e11" } modify_external_catalogs: true
I20251211 08:57:57.743180 14012 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_27021f5f53d9422787bd29bc1df43e11" } modify_external_catalogs: true
I20251211 08:57:57.743314 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6d15b83379564d4cbcf8329733fded8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:57.744745 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d15b83379564d4cbcf8329733fded8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:57.746928 15950 raft_consensus.cc:493] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:57.747393 15950 raft_consensus.cc:515] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:57.748229 16093 raft_consensus.cc:515] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.749830 16093 leader_election.cc:290] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:57.750985 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9d63dc89d7e4fdc8615fe78e0882be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.751994 14439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3403116517f45c59f87fa5bda0b4071" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:57.752523 14439 raft_consensus.cc:3060] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:57.754871 15950 leader_election.cc:290] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:57.755497 14231 leader_election.cc:343] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 8e05628011634149b17d298c54074d00
I20251211 08:57:57.756561 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f72e4f600b40d6a762103cd70d9e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.759166 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e147eeb943a2452fbc507b5e5a955357. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.763545 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b73a1f96140455eb9ff536eb06973cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.768864 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet eccc727462b447489372da09ea828ac2
I20251211 08:57:57.768379 14722 consensus_queue.cc:260] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.769580 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80853e3a1ca24fcdb889aee9dd6619f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.770485 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet eccc727462b447489372da09ea828ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:57.779435 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_44adccbd58af4364b3956596d908c42f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:57.780519 14439 raft_consensus.cc:2468] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
W20251211 08:57:57.780809 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44adccbd58af4364b3956596d908c42f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:57.781898 14231 leader_election.cc:304] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:57:57.788857 15941 tablet_replica.cc:333] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:57.793090 15902 consensus_queue.cc:1048] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:57:57.796905 15030 raft_consensus.cc:359] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.797628 15030 raft_consensus.cc:385] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:57.797884 15030 raft_consensus.cc:740] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:57.798969 14579 tablet_service.cc:1552] Processing DeleteTablet for tablet eccc727462b447489372da09ea828ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:57:57.799765 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 137271us
W20251211 08:57:57.800347 14542 consensus_peers.cc:597] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45c7d841bd3643a799a8b96e82d6d675. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:57.798446 15030 consensus_queue.cc:260] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.801522 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_56e0be458512461a81a0341c520b616d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:57.803017 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56e0be458512461a81a0341c520b616d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:57.809267 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffaed0b6158a43b781a2541c745a1c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.809850 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e70de61c3dbb440eb5608abb114d37fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.810276 16093 raft_consensus.cc:2804] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:57.810757 16093 raft_consensus.cc:697] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:57:57.811691 16093 consensus_queue.cc:237] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.815071 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5872a08932b549948ff00504e8962ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:57.839138 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a197b5bcd78844abae3b6d6e25017d53" } new_table_name: "default.Table_655bca8d0e074134b47d51c12cd4835a" modify_external_catalogs: true
I20251211 08:57:57.866469 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_0db322b6eed54744bc0f600c81c4d1c5" } new_table_name: "default.Table_a7ed3d0a6ce0464893bc9f5bb3bf6851" modify_external_catalogs: true
W20251211 08:57:57.868055 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76f72e4f600b40d6a762103cd70d9e85 consensus state has no leader
I20251211 08:57:57.870644 15942 raft_consensus.cc:493] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:57.871097 15942 raft_consensus.cc:515] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.873492 15942 leader_election.cc:290] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:57.874236 15941 raft_consensus.cc:2243] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:57.874876 15941 raft_consensus.cc:2272] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:57.886122 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48575fa3f45349b9887bfc1cdfebc958 consensus state has no leader
I20251211 08:57:57.904803 14904 raft_consensus.cc:2243] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:57.913769 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b96aa9120021443caa9cf1d91e71e6b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:57.915292 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b96aa9120021443caa9cf1d91e71e6b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:57.920835 14260 tablet_service.cc:1552] Processing DeleteTablet for tablet c341b762afd8440bb7198ca134c8dffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:37 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:57.934188 14905 ts_tablet_manager.cc:1929] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:57.934695 14905 log.cc:1199] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/4ab48f6288494005ad42208e2e97ddf2
I20251211 08:57:57.941941 14904 raft_consensus.cc:2272] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:57.942812 14722 ts_tablet_manager.cc:1434] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.228s user 0.003s sys 0.004s
I20251211 08:57:57.943924 14722 tablet_bootstrap.cc:492] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:57.943967 14904 ts_tablet_manager.cc:1916] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:57.947965 14272 tablet_service.cc:1552] Processing DeleteTablet for tablet ccce5a446ac9460c9c9b7de72d875d95 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ce1ef2236f54feab5c38bc401c126e8) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:57.963380 14905 ts_tablet_manager.cc:1950] T 4ab48f6288494005ad42208e2e97ddf2 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:57.876873 15941 ts_tablet_manager.cc:1916] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:57.887406 14714 tablet_bootstrap.cc:492] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:57.973440 14714 ts_tablet_manager.cc:1403] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.291s user 0.009s sys 0.008s
W20251211 08:57:57.975448 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 111adfd1ac1a42bdafa9416594e7e932 consensus state has no leader
I20251211 08:57:57.975523 14714 raft_consensus.cc:359] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.976027 14714 raft_consensus.cc:385] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:57.976279 14714 raft_consensus.cc:740] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:57.976476 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_44adccbd58af4364b3956596d908c42f" } modify_external_catalogs: true
I20251211 08:57:57.976926 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_44adccbd58af4364b3956596d908c42f" } modify_external_catalogs: true
I20251211 08:57:57.976814 14714 consensus_queue.cc:260] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:57.980605 14714 ts_tablet_manager.cc:1434] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251211 08:57:57.981447 14714 tablet_bootstrap.cc:492] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:57.988919 15030 ts_tablet_manager.cc:1434] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.283s user 0.006s sys 0.000s
I20251211 08:57:57.994915 15941 ts_tablet_manager.cc:1929] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:57.995200 15941 log.cc:1199] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/c341b762afd8440bb7198ca134c8dffe
I20251211 08:57:57.997570 14008 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 2c667452690c40738eb3f7f34ea23438
I20251211 08:57:58.001603 14714 tablet_bootstrap.cc:654] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:58.005918 15941 ts_tablet_manager.cc:1950] T c341b762afd8440bb7198ca134c8dffe P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:58.007802 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_5e20e9bb13c4482b8b1a3dca8ef768c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:58.008365 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 505) took 2096 ms. Trace:
W20251211 08:57:58.008718 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 208065us
I20251211 08:57:58.008591 15941 rpcz_store.cc:276] 1211 08:57:55.911541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.911629 (+ 88us) service_pool.cc:225] Handling call
1211 08:57:57.788669 (+1877040us) ts_tablet_manager.cc:1151] Deleting tablet c341b762afd8440bb7198ca134c8dffe
1211 08:57:57.788721 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.988776 (+200055us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.994724 (+ 5948us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.008320 (+ 13596us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":84539,"tablet-delete.queue_time_us":1876515}
W20251211 08:57:58.011612 14542 leader_election.cc:343] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 07aef1eea7c04d9cbf35c82e45df7a16
I20251211 08:57:58.012354 14542 leader_election.cc:304] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:58.012615 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet c341b762afd8440bb7198ca134c8dffe (table default.Table_96d05b744df34836be68b4841e6c2a2b [id=86d8260b23ca4945acd9a70054b77c26]) successfully deleted
I20251211 08:57:58.013763 16044 raft_consensus.cc:2804] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:58.014142 16044 raft_consensus.cc:697] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:58.016213 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 506) took 2100 ms. Trace:
I20251211 08:57:58.016508 15941 rpcz_store.cc:276] 1211 08:57:55.915698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.915848 (+ 150us) service_pool.cc:225] Handling call
1211 08:57:58.009356 (+2093508us) ts_tablet_manager.cc:1151] Deleting tablet 785a5edbda984467b897ee1bbaf1430c
1211 08:57:58.015939 (+ 6583us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.016168 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2092978}
I20251211 08:57:58.017540 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 510) took 1240 ms. Trace:
I20251211 08:57:58.018546 14439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51c3f5eaef784e868e565f81bd51a5e4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:58.019106 14439 raft_consensus.cc:2468] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
W20251211 08:57:58.021622 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 785a5edbda984467b897ee1bbaf1430c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 785a5edbda984467b897ee1bbaf1430c already in progress: creating tablet
I20251211 08:57:58.017843 15941 rpcz_store.cc:276] 1211 08:57:56.776687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:56.776810 (+ 123us) service_pool.cc:225] Handling call
1211 08:57:58.017223 (+1240413us) ts_tablet_manager.cc:1151] Deleting tablet 1b2445f66be54d3fbdc3be964b9a9132
1211 08:57:58.017253 (+ 30us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.017497 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1239966}
W20251211 08:57:58.031219 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 1b2445f66be54d3fbdc3be964b9a9132 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b2445f66be54d3fbdc3be964b9a9132 already in progress: creating tablet
I20251211 08:57:58.033151 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f7dd35033be14c598ae0f9801daa1c4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:58.034459 14273 tablet_service.cc:1552] Processing DeleteTablet for tablet 15595c5a432047de873c2647b78838d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:58.036276 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 511) took 1238 ms. Trace:
W20251211 08:57:58.039072 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7dd35033be14c598ae0f9801daa1c4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:58.039109 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e20e9bb13c4482b8b1a3dca8ef768c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.047492 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_61cd87d7f57741dd997ee9bac6835add"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:58.048797 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61cd87d7f57741dd997ee9bac6835add in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.038618 15941 rpcz_store.cc:276] 1211 08:57:56.797996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:56.798116 (+ 120us) service_pool.cc:225] Handling call
1211 08:57:58.034554 (+1236438us) ts_tablet_manager.cc:1151] Deleting tablet 225438e70aae424a9671aaceb3ee0ed6
1211 08:57:58.034635 (+ 81us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.035989 (+ 1354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1232374}
W20251211 08:57:58.055266 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 225438e70aae424a9671aaceb3ee0ed6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 225438e70aae424a9671aaceb3ee0ed6 already in progress: creating tablet
I20251211 08:57:58.055262 14274 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c0f8d7dd90f4beaae674fe827d37e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:58.057081 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 522) took 1137 ms. Trace:
I20251211 08:57:58.066843 15941 rpcz_store.cc:276] 1211 08:57:56.919271 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:56.919475 (+ 204us) service_pool.cc:225] Handling call
1211 08:57:58.056791 (+1137316us) ts_tablet_manager.cc:1151] Deleting tablet 476eee1606e74832abf928dbaafc844d
1211 08:57:58.056825 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.057035 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1135159}
W20251211 08:57:58.068022 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 476eee1606e74832abf928dbaafc844d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 476eee1606e74832abf928dbaafc844d already in progress: creating tablet
W20251211 08:57:58.069480 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet e9905227c0a844d199362a442e7d630c because the tablet was not found. No further retry: Not found: Tablet not found: e9905227c0a844d199362a442e7d630c
I20251211 08:57:58.069579 14722 tablet_bootstrap.cc:654] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:58.070231 15941 tablet_replica.cc:333] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:58.072487 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet da52cace2aab4f9dacef18ec5fa56aaf (table default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]) successfully deleted
I20251211 08:57:58.073779 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e46c6e53d82c408980e7e825ef7a1084" } new_table_name: "default.Table_c9d6152b39d84c43b010c20703af60ac" modify_external_catalogs: true
I20251211 08:57:58.071080 15941 raft_consensus.cc:2243] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:58.077363 15941 raft_consensus.cc:2272] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:58.081565 15941 ts_tablet_manager.cc:1916] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:58.105643 16044 consensus_queue.cc:237] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:57:58.120080 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 07099552b546444bab4d66238dd6d146 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07099552b546444bab4d66238dd6d146 already in progress: creating tablet
I20251211 08:57:58.122304 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7bce90f465c249f9a5ab2b250fb5610b" } new_table_name: "default.Table_3c68727354f7437e8d73d559f04cd3ea" modify_external_catalogs: true
I20251211 08:57:58.130133 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51c3f5eaef784e868e565f81bd51a5e4" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.130620 14607 raft_consensus.cc:2468] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:58.132252 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2f095b59ecf44ffabaa0ac22212938b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.132823 14603 raft_consensus.cc:2468] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:58.133592 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3403116517f45c59f87fa5bda0b4071" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.134138 14605 raft_consensus.cc:2468] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:58.135380 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "033c2c016f8f4c01adc8bdbf2885e394" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:58.136703 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet ca5eebf2ce3848c79ee2fc21ad2b1486 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca5eebf2ce3848c79ee2fc21ad2b1486 already in progress: creating tablet
I20251211 08:57:58.141588 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c04583bef74d84a49cdcd8a1305a88" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.148054 14274 tablet_service.cc:1552] Processing DeleteTablet for tablet eccc727462b447489372da09ea828ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:58.152801 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_cc7fd4312d5d4dc7807e350123a637e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:58.154244 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc7fd4312d5d4dc7807e350123a637e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.156847 14714 tablet_bootstrap.cc:492] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:58.157248 14714 ts_tablet_manager.cc:1403] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.176s user 0.006s sys 0.004s
W20251211 08:57:58.165390 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet b49ea29851d94622b4299e93804a0578 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b49ea29851d94622b4299e93804a0578 already in progress: creating tablet
W20251211 08:57:58.167073 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 143f20f5c37948068ff2fb0d92f0f34b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 143f20f5c37948068ff2fb0d92f0f34b already in progress: creating tablet
I20251211 08:57:58.169555 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 405) took 9095 ms. Trace:
I20251211 08:57:58.169981 14905 rpcz_store.cc:276] 1211 08:57:49.073693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.079082 (+ 5389us) service_pool.cc:225] Handling call
1211 08:57:57.458427 (+8379345us) ts_tablet_manager.cc:1151] Deleting tablet 4ab48f6288494005ad42208e2e97ddf2
1211 08:57:57.466072 (+ 7645us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:57.805112 (+339040us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:57.933963 (+128851us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.169482 (+235519us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":225536,"tablet-delete.queue_time_us":8377637}
I20251211 08:57:58.170981 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 49015c5d4f2b4d07b547a9df57c992eb (table default.Table_73091d56834e4e78b870257988c87c57 [id=d8f07591554543f2880445cb636da15a]) successfully deleted
I20251211 08:57:58.180547 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 4ab48f6288494005ad42208e2e97ddf2 (table default.Table_cd9ec56dc08c48779e88a778d2917bb0 [id=658ce49773af41519ab660f3bc29975b]) successfully deleted
I20251211 08:57:58.176494 15030 tablet_bootstrap.cc:492] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
W20251211 08:57:58.185020 14576 consensus_meta.cc:287] T 76f72e4f600b40d6a762103cd70d9e85 P c5e624bcfd154cd586ef1dedb9940063: Time spent flushing consensus metadata: real 0.528s user 0.007s sys 0.004s
I20251211 08:57:58.186296 14722 tablet_bootstrap.cc:492] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:58.187647 14722 ts_tablet_manager.cc:1403] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.244s user 0.010s sys 0.003s
W20251211 08:57:58.188339 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2ee6cab1d1742e08323842ad9c69b81 consensus state has no leader
I20251211 08:57:58.191658 14714 raft_consensus.cc:359] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.192243 14714 raft_consensus.cc:385] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:58.192485 14714 raft_consensus.cc:740] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:58.200848 14284 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c04583bef74d84a49cdcd8a1305a88" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:58.201622 14284 raft_consensus.cc:2468] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:58.203367 14714 consensus_queue.cc:260] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:57:58.205840 14544 consensus_peers.cc:597] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45c7d841bd3643a799a8b96e82d6d675. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:58.202816 14722 raft_consensus.cc:359] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.206485 14722 raft_consensus.cc:385] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:58.206890 14722 raft_consensus.cc:740] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:58.207481 14722 consensus_queue.cc:260] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.210182 14289 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3403116517f45c59f87fa5bda0b4071" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:58.211650 14291 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2f095b59ecf44ffabaa0ac22212938b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:58.214824 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51c3f5eaef784e868e565f81bd51a5e4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:58.215039 14586 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42846 (request call id 492) took 1045 ms. Trace:
I20251211 08:57:58.215340 14295 raft_consensus.cc:2468] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:58.216368 14293 raft_consensus.cc:3060] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:58.219530 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41ef5bd335a44da39f2549fd55a048f9 consensus state has no leader
I20251211 08:57:58.219867 15030 tablet_bootstrap.cc:654] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:58.221185 14544 consensus_peers.cc:597] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:58.221549 14008 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_1df60c18431f4fcbb050908676608f64" } modify_external_catalogs: true
I20251211 08:57:58.215337 14586 rpcz_store.cc:276] 1211 08:57:57.170037 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.205386 (+ 35349us) service_pool.cc:225] Handling call
1211 08:57:57.207317 (+ 1931us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1211 08:57:57.207382 (+ 65us) ts_tablet_manager.cc:682] Creating new metadata...
1211 08:57:57.491388 (+284006us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.208023 (+716635us) tablet_replica.cc:206] Creating consensus instance
1211 08:57:58.214987 (+ 6964us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":569209}
I20251211 08:57:58.222026 14008 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_1df60c18431f4fcbb050908676608f64" } modify_external_catalogs: true
I20251211 08:57:58.217005 14905 tablet_replica.cc:333] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:58.225771 15941 ts_tablet_manager.cc:1929] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:58.226125 15941 log.cc:1199] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/2a1b011cb7c647f79bb16ce1c92bcff5
I20251211 08:57:58.226929 14905 raft_consensus.cc:2243] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:58.228251 14905 raft_consensus.cc:2272] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:58.233510 14905 ts_tablet_manager.cc:1916] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:58.236485 14544 leader_election.cc:304] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:57:58.244027 16112 tablet.cc:1722] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:58.251586 14904 ts_tablet_manager.cc:1929] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:58.260564 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f8b62284383b447f88f4ecaa25393a1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:58.261328 16092 raft_consensus.cc:2804] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:57:58.261680 16092 raft_consensus.cc:493] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:57:58.261979 16092 raft_consensus.cc:3060] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Advancing to term 2
W20251211 08:57:58.264709 14231 consensus_peers.cc:597] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 080d1700df4a4df09acda65c6d021d96. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:58.226732 15941 ts_tablet_manager.cc:1950] T 2a1b011cb7c647f79bb16ce1c92bcff5 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:58.268810 15941 tablet_replica.cc:333] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:58.269451 14722 ts_tablet_manager.cc:1434] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.080s user 0.005s sys 0.000s
I20251211 08:57:58.269477 15941 raft_consensus.cc:2243] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:58.269869 15941 raft_consensus.cc:2272] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:57:58.270747 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 44486.3us
W20251211 08:57:58.271624 14385 leader_election.cc:343] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:58.272118 14385 leader_election.cc:343] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:58.272630 14385 leader_election.cc:304] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
W20251211 08:57:58.273427 14385 leader_election.cc:343] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:58.274072 16092 raft_consensus.cc:515] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:58.275254 16124 raft_consensus.cc:2804] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:58.275565 16124 raft_consensus.cc:493] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:58.275846 16124 raft_consensus.cc:3060] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:58.281699 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8b62284383b447f88f4ecaa25393a1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.282588 14714 ts_tablet_manager.cc:1434] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.125s user 0.007s sys 0.000s
I20251211 08:57:58.285120 14714 tablet_bootstrap.cc:492] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:58.251952 14904 log.cc:1199] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/8e05628011634149b17d298c54074d00
I20251211 08:57:58.286521 14904 ts_tablet_manager.cc:1950] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:58.288259 15941 ts_tablet_manager.cc:1916] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:58.288329 15902 consensus_queue.cc:1048] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:58.298717 14544 leader_election.cc:304] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:58.299477 16099 raft_consensus.cc:2804] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:58.299832 16099 raft_consensus.cc:493] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:58.300168 16099 raft_consensus.cc:3060] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:58.308604 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_756c31646dff4edfba939b6fed9f48c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:58.310173 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_756c31646dff4edfba939b6fed9f48c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.311925 14012 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_dfdb7a58dd4c42b9a73c75fd44f74fd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:58.313304 14012 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfdb7a58dd4c42b9a73c75fd44f74fd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:58.326231 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 189644us
I20251211 08:57:58.330281 14382 leader_election.cc:304] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:58.331008 15942 raft_consensus.cc:2804] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:58.331492 15942 raft_consensus.cc:493] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:58.331903 15942 raft_consensus.cc:3060] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:58.338723 15942 raft_consensus.cc:515] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.340656 15942 leader_election.cc:290] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:58.353029 16092 leader_election.cc:290] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:58.353682 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 55056.6us
I20251211 08:57:58.306416 16099 raft_consensus.cc:515] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.367107 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 461) took 2572 ms. Trace:
I20251211 08:57:58.367424 14904 rpcz_store.cc:276] 1211 08:57:55.794384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.794483 (+ 99us) service_pool.cc:225] Handling call
1211 08:57:57.461252 (+1666769us) ts_tablet_manager.cc:1151] Deleting tablet 8e05628011634149b17d298c54074d00
1211 08:57:57.461284 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.065567 (+604283us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.251376 (+185809us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.367041 (+115665us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":256942,"tablet-delete.queue_time_us":1666283}
I20251211 08:57:58.368535 14904 tablet_replica.cc:333] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:58.369200 14904 raft_consensus.cc:2243] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:58.369627 14904 raft_consensus.cc:2272] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:58.367441 14722 tablet_bootstrap.cc:492] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:58.370960 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 8e05628011634149b17d298c54074d00 (table default.Table_885d3fb0ab5f4d0a82972d94a34c2a88 [id=18d1df6385a24d26bfb44309aa58eaa9]) successfully deleted
I20251211 08:57:58.374650 14904 ts_tablet_manager.cc:1916] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:58.376343 14722 tablet_bootstrap.cc:654] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:58.371717 15030 tablet_bootstrap.cc:492] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:58.388788 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfb3a2b8f8f64c12ab590ffc0be7ace3" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.390034 16099 leader_election.cc:290] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:57:58.397599 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 123434us
I20251211 08:57:58.397567 14012 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_4d9c098623e84435972e3c148a08b83e" } new_table_name: "default.Table_6f0fe9f7f8ca4ee69de561cba58b14d4" modify_external_catalogs: true
I20251211 08:57:58.399062 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 2a1b011cb7c647f79bb16ce1c92bcff5 (table default.Table_03bee763242749cfa5fb0fe7d7e7fb7f [id=209008ecbdd4441989221f0f956a87b6]) successfully deleted
I20251211 08:57:58.394297 16131 raft_consensus.cc:493] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:58.400725 16131 raft_consensus.cc:515] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.402618 16131 leader_election.cc:290] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:58.403504 14904 ts_tablet_manager.cc:1929] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:58.403887 14904 log.cc:1199] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/07aef1eea7c04d9cbf35c82e45df7a16
I20251211 08:57:58.404096 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51c3f5eaef784e868e565f81bd51a5e4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:58.404866 14441 raft_consensus.cc:2393] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c5e624bcfd154cd586ef1dedb9940063 in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:57:58.405110 16137 tablet.cc:1722] T f5c5b1aebde94dea81cb37a6a6998070 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251211 08:57:58.405962 14542 consensus_peers.cc:597] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e05628011634149b17d298c54074d00. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:58.406081 14904 ts_tablet_manager.cc:1950] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:57:58.390554 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 60377.2us
W20251211 08:57:58.425330 16070 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 12660783061858645059) tablet 00000000000000000000000000000000 cur holder 0x7b5800045300
I20251211 08:57:58.427333 14714 tablet_bootstrap.cc:654] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:58.429517 16124 raft_consensus.cc:515] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.431795 16124 leader_election.cc:290] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:58.435436 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_5bff583c9d95495aa8ac02605828fa4d" } new_table_name: "default.Table_7be0669429a44a41919ecc13aa32caec" modify_external_catalogs: true
W20251211 08:57:58.452837 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 46997.6us
W20251211 08:57:58.456429 14542 consensus_peers.cc:597] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9905227c0a844d199362a442e7d630c. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:58.462787 14011 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:57:58.449808 14382 leader_election.cc:304] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:58.467469 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet f2f095b59ecf44ffabaa0ac22212938b with delete_type TABLET_DATA_DELETED (replaced by tablet b3ad595efe5241e8a4f40b16be2ccd63) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:58.466745 16131 raft_consensus.cc:2804] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:58.468292 16131 raft_consensus.cc:493] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:58.468571 16131 raft_consensus.cc:3060] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:58.470752 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_0275161db5ae45b6950bd8b95723b20b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:58.472218 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0275161db5ae45b6950bd8b95723b20b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.473677 14413 tablet_service.cc:1505] Processing CreateTablet for tablet 22c89da854ef4396a1e1f4173020589b (DEFAULT_TABLE table=default.Table_b5cff0352c23472291dd9150a225a639 [id=5ea5300fb27749a59229d43e3423848a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:58.474591 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c89da854ef4396a1e1f4173020589b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:58.475013 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 455) took 2682 ms. Trace:
I20251211 08:57:58.475069 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51c3f5eaef784e868e565f81bd51a5e4" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:58.475615 14296 raft_consensus.cc:3060] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:58.476329 14382 leader_election.cc:304] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:57:58.475353 14904 rpcz_store.cc:276] 1211 08:57:55.792904 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.819011 (+ 26107us) service_pool.cc:225] Handling call
1211 08:57:58.368401 (+2549390us) ts_tablet_manager.cc:1151] Deleting tablet 07aef1eea7c04d9cbf35c82e45df7a16
1211 08:57:58.368435 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.384200 (+ 15765us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.403283 (+ 19083us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.474960 (+ 71677us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2547218}
W20251211 08:57:58.477260 14382 leader_election.cc:343] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:58.477598 16128 raft_consensus.cc:2764] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251211 08:57:58.436168 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3403116517f45c59f87fa5bda0b4071" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.436873 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3403116517f45c59f87fa5bda0b4071" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:58.438915 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e05628011634149b17d298c54074d00" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.441784 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07aef1eea7c04d9cbf35c82e45df7a16" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.480440 14607 raft_consensus.cc:2393] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31404d5f58e548519239569b150d975b in current term 2: Already voted for candidate c5e624bcfd154cd586ef1dedb9940063 in this term.
I20251211 08:57:58.479768 14599 raft_consensus.cc:3060] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:58.481938 14542 leader_election.cc:343] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 07aef1eea7c04d9cbf35c82e45df7a16
I20251211 08:57:58.482653 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 07aef1eea7c04d9cbf35c82e45df7a16 (table default.Table_d40cb8a6b4974e2e9ff982e893a2c552 [id=d6acd0e03f2743349b345927f5ce7a6d]) successfully deleted
W20251211 08:57:58.485774 14012 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 827b6a73ea5a4462ae58f36f12919b0c consensus state has no leader
I20251211 08:57:58.487233 14599 raft_consensus.cc:2468] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:57:58.487480 15941 ts_tablet_manager.cc:1929] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:58.487812 15941 log.cc:1199] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/f785a3cc2e8145eebae7e9c2885b0b1e
I20251211 08:57:58.488444 15941 ts_tablet_manager.cc:1950] T f785a3cc2e8145eebae7e9c2885b0b1e P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:58.493214 14598 raft_consensus.cc:2376] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 31404d5f58e548519239569b150d975b in term 1. Re-sending same reply.
I20251211 08:57:58.417586 15030 ts_tablet_manager.cc:1403] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.241s user 0.014s sys 0.000s
I20251211 08:57:58.494901 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 22c89da854ef4396a1e1f4173020589b (DEFAULT_TABLE table=default.Table_b5cff0352c23472291dd9150a225a639 [id=5ea5300fb27749a59229d43e3423848a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:58.495812 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c89da854ef4396a1e1f4173020589b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:58.497740 14266 tablet_service.cc:1505] Processing CreateTablet for tablet dd68012b5032448d9709caeb6ea98a2a (DEFAULT_TABLE table=default.Table_210056e85c404f0082fe3bc88c22c5bb [id=3a0b5df91cb64a82960635cbdf0dd38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:58.498672 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd68012b5032448d9709caeb6ea98a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:58.498580 14419 tablet_service.cc:1505] Processing CreateTablet for tablet dd68012b5032448d9709caeb6ea98a2a (DEFAULT_TABLE table=default.Table_210056e85c404f0082fe3bc88c22c5bb [id=3a0b5df91cb64a82960635cbdf0dd38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:58.499648 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd68012b5032448d9709caeb6ea98a2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:58.503414 14011 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 61) took 1043 ms (client timeout 1000 ms). Trace:
W20251211 08:57:58.503787 14011 rpcz_store.cc:269] 1211 08:57:57.459822 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.459948 (+ 126us) service_pool.cc:225] Handling call
1211 08:57:58.503347 (+1043399us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":4}
I20251211 08:57:58.509294 16137 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251211 08:57:58.512110 14576 tablet_service.cc:1505] Processing CreateTablet for tablet dd68012b5032448d9709caeb6ea98a2a (DEFAULT_TABLE table=default.Table_210056e85c404f0082fe3bc88c22c5bb [id=3a0b5df91cb64a82960635cbdf0dd38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:58.513023 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd68012b5032448d9709caeb6ea98a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:58.512975 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_5467e261c9f5488f9084d51d89721df3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:58.516142 14414 tablet_service.cc:1505] Processing CreateTablet for tablet 10d8b7117acd41a9a6cc356ed51bb6e8 (DEFAULT_TABLE table=default.Table_a4dd5045e2db4928b29f19e423ba3188 [id=92f301dfe9bf45efa44c96ceea930854]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:58.517184 14414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d8b7117acd41a9a6cc356ed51bb6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:58.502555 16131 raft_consensus.cc:515] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.521365 14008 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 020230ff08ed42179a28efbb21613a0f
I20251211 08:57:58.477281 14904 tablet_replica.cc:333] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:57:58.523517 14904 raft_consensus.cc:2243] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:58.523779 14582 tablet_service.cc:1552] Processing DeleteTablet for tablet 020230ff08ed42179a28efbb21613a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:58 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:58.527659 14256 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:46838 (request call id 71) took 1307 ms. Trace:
W20251211 08:57:58.535370 14005 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 785ccfbc31b74ca6969bcf4020b0520c consensus state has no leader
I20251211 08:57:58.538645 15030 raft_consensus.cc:359] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.548060 15030 raft_consensus.cc:385] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:58.548389 15030 raft_consensus.cc:740] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:58.549191 15030 consensus_queue.cc:260] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.551630 15030 ts_tablet_manager.cc:1434] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20251211 08:57:58.560410 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 3aed0129e9164a779b73994b8b37426c
I20251211 08:57:58.586071 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_07dddc710b174f5ca438600d9fe40b91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:58.529790 14904 raft_consensus.cc:2272] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:58.530355 14286 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111adfd1ac1a42bdafa9416594e7e932" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:58.471010 15950 raft_consensus.cc:493] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:58.531543 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet 020230ff08ed42179a28efbb21613a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:58 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:58.532222 16096 raft_consensus.cc:493] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:58.533941 14261 tablet_service.cc:1505] Processing CreateTablet for tablet 22c89da854ef4396a1e1f4173020589b (DEFAULT_TABLE table=default.Table_b5cff0352c23472291dd9150a225a639 [id=5ea5300fb27749a59229d43e3423848a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:58.536408 14288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c04583bef74d84a49cdcd8a1305a88" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:57:58.537554 14231 consensus_peers.cc:597] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f785a3cc2e8145eebae7e9c2885b0b1e. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:57:58.562912 14268 tablet_service.cc:1552] Processing DeleteTablet for tablet 3aed0129e9164a779b73994b8b37426c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:57:58.590109 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07dddc710b174f5ca438600d9fe40b91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.585520 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_415671e3e703478eb0cf348eb2943243"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:58.592331 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_415671e3e703478eb0cf348eb2943243 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.562603 14582 tablet_service.cc:1552] Processing DeleteTablet for tablet 3aed0129e9164a779b73994b8b37426c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:58.563635 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51c3f5eaef784e868e565f81bd51a5e4" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:58.605069 14598 raft_consensus.cc:2393] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b276f77f87254a5d909c5ab940a39256 in current term 1: Already voted for candidate c5e624bcfd154cd586ef1dedb9940063 in this term.
I20251211 08:57:58.582013 14296 raft_consensus.cc:2468] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:58.580822 16141 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251211 08:57:58.592332 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111adfd1ac1a42bdafa9416594e7e932" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.593217 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c04583bef74d84a49cdcd8a1305a88" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:58.570562 16112 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:42846 (request call id 476) took 2927 ms. Trace:
I20251211 08:57:58.612986 14607 raft_consensus.cc:3060] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:58.612957 16112 rpcz_store.cc:276] 1211 08:57:55.643064 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.643201 (+ 137us) service_pool.cc:225] Handling call
1211 08:57:58.570503 (+2927302us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:56.243982 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1211 08:57:56.259478 (+ 15496us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1211 08:57:56.259515 (+ 37us) alter_schema_op.cc:66] Acquired schema lock
1211 08:57:56.259579 (+ 64us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1211 08:57:56.259702 (+ 123us) alter_schema_op.cc:128] START. Timestamp: 1765443476259651
1211 08:57:56.259723 (+ 21us) op_driver.cc:348] REPLICATION: starting
1211 08:57:56.260322 (+ 599us) log.cc:844] Serialized 173 byte log entry
1211 08:57:57.987740 (+1727418us) op_driver.cc:464] REPLICATION: finished
1211 08:57:58.243948 (+256208us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1211 08:57:58.245188 (+ 1240us) log.cc:844] Serialized 5 byte log entry
1211 08:57:58.569006 (+323818us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.569711 (+ 705us) log.cc:844] Serialized 14 byte log entry
1211 08:57:58.570379 (+ 668us) alter_schema_op.cc:73] Released schema lock
1211 08:57:58.570420 (+ 41us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":256053,"prepare.queue_time_us":575133,"prepare.run_cpu_time_us":9691,"prepare.run_wall_time_us":145017,"replication_time_us":1727914,"thread_start_us":926269,"threads_started":4,"wal-append.queue_time_us":192150,"wal-append.run_cpu_time_us":1757,"wal-append.run_wall_time_us":1289164}]]}
I20251211 08:57:58.619788 14607 raft_consensus.cc:2468] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:58.620359 14905 ts_tablet_manager.cc:1929] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:58.627990 14905 log.cc:1199] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/f2f095b59ecf44ffabaa0ac22212938b
I20251211 08:57:58.628750 14905 ts_tablet_manager.cc:1950] T f2f095b59ecf44ffabaa0ac22212938b P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:58.588243 15950 raft_consensus.cc:515] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.630863 15950 leader_election.cc:290] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:58.588936 14904 ts_tablet_manager.cc:1916] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:58.589111 14261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c89da854ef4396a1e1f4173020589b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:58.589367 14288 raft_consensus.cc:3060] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:57:58.589534 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 52126.8us
I20251211 08:57:58.602653 14722 tablet_bootstrap.cc:492] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:58.639089 14722 ts_tablet_manager.cc:1403] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.272s user 0.005s sys 0.012s
I20251211 08:57:58.641623 14722 raft_consensus.cc:359] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.642359 14722 raft_consensus.cc:385] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:58.642637 14722 raft_consensus.cc:740] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:58.643445 14722 consensus_queue.cc:260] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.646456 14722 ts_tablet_manager.cc:1434] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20251211 08:57:58.651324 14722 tablet_bootstrap.cc:492] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:58.588544 16096 raft_consensus.cc:515] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:58.657716 16096 leader_election.cc:290] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:57:58.663125 14231 leader_election.cc:343] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 2a1b011cb7c647f79bb16ce1c92bcff5
I20251211 08:57:58.664063 15030 tablet_bootstrap.cc:492] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:58.666055 14722 tablet_bootstrap.cc:654] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:58.666497 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 408) took 9558 ms. Trace:
I20251211 08:57:58.669483 14905 rpcz_store.cc:276] 1211 08:57:49.108464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.115759 (+ 7295us) service_pool.cc:225] Handling call
1211 08:57:58.207538 (+9091779us) ts_tablet_manager.cc:1151] Deleting tablet f2f095b59ecf44ffabaa0ac22212938b
1211 08:57:58.216900 (+ 9362us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.442102 (+225202us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.620153 (+178051us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.666441 (+ 46288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9080359}
I20251211 08:57:58.670306 14013 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:57:58.670572 14905 tablet_replica.cc:333] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:57:58.671304 14231 leader_election.cc:343] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: dd14de369e084828afb8a924c46c649b
I20251211 08:57:58.671451 14905 raft_consensus.cc:2243] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Raft consensus shutting down.
I20251211 08:57:58.674008 15902 raft_consensus.cc:493] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:58.674423 15902 raft_consensus.cc:515] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.677302 14013 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_31fcfbed4a2c4e8aa05fb42c261bf1d5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 2 got 0
I20251211 08:57:58.677435 15902 leader_election.cc:290] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:58.677531 14722 tablet_bootstrap.cc:492] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:58.677893 14722 ts_tablet_manager.cc:1403] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.001s
I20251211 08:57:58.679298 14905 raft_consensus.cc:2272] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:58.680045 14722 raft_consensus.cc:359] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.680629 14722 raft_consensus.cc:385] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:58.680853 14722 raft_consensus.cc:740] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:58.681375 14722 consensus_queue.cc:260] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.683045 14905 ts_tablet_manager.cc:1916] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:58.683918 14288 raft_consensus.cc:2468] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:58.684862 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet f2f095b59ecf44ffabaa0ac22212938b (table default.Table_fee4bd06cd7e4df08fa4bafdd291a3d4 [id=e754a76685124291a154d2a35952abd3]) successfully deleted
I20251211 08:57:58.686890 15030 tablet_bootstrap.cc:654] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:58.678409 14013 catalog_manager.cc:5654] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20251211 08:57:58.688551 14013 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (table default.Table_321194b0f71d457e81e14c810650b486 [id=f74e074787144a21ad31d4d89cc509c9]). Expected version 1 got 0
I20251211 08:57:58.688647 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_c264c2b8de9a4f97b223eb3bfef06afb" } new_table_name: "default.Table_032686fd837d4292bd6c95749e67b5de" modify_external_catalogs: true
I20251211 08:57:58.689699 14013 catalog_manager.cc:5654] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } }
I20251211 08:57:58.586401 14256 rpcz_store.cc:276] 1211 08:57:57.149659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.302050 (+152391us) service_pool.cc:225] Handling call
1211 08:57:58.086469 (+784419us) spinlock_profiling.cc:243] Waited 6.42 ms on lock 0x7b54001a0fa8. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f146046c1 00007f0f146902d2 00007f0f1529d1e0 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:57:58.457199 (+370730us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14819328}
I20251211 08:57:58.694633 14722 ts_tablet_manager.cc:1434] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20251211 08:57:58.695101 14013 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_7ee755c24982464a9d02be456caa647e [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 2 got 0
I20251211 08:57:58.695606 14013 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:57:58.524374 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet 020230ff08ed42179a28efbb21613a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:58 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:58.514730 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5467e261c9f5488f9084d51d89721df3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.563571 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 3aed0129e9164a779b73994b8b37426c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:57 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:58.477591 14382 leader_election.cc:304] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:58.529973 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 531) took 1200 ms. Trace:
I20251211 08:57:58.704236 15941 rpcz_store.cc:276] 1211 08:57:57.289964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.290083 (+ 119us) service_pool.cc:225] Handling call
1211 08:57:58.268692 (+978609us) ts_tablet_manager.cc:1151] Deleting tablet f785a3cc2e8145eebae7e9c2885b0b1e
1211 08:57:58.268721 (+ 29us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.392775 (+124054us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.487271 (+ 94496us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.490178 (+ 2907us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":978109}
W20251211 08:57:58.597781 14385 leader_election.cc:343] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:58.604777 14714 tablet_bootstrap.cc:492] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:58.705590 14714 ts_tablet_manager.cc:1403] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.421s user 0.015s sys 0.020s
W20251211 08:57:58.706028 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 108458us
I20251211 08:57:58.706707 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet f785a3cc2e8145eebae7e9c2885b0b1e (table default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]) successfully deleted
I20251211 08:57:58.706816 14013 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 578404732eef4e51b66b6aebffa87cdb (table default.Table_48a06c2ab96d42feb95c8813fb635c80 [id=26c0a9d9b1f84372b18a54f045eb768c]). Expected version 6 got 5
W20251211 08:57:58.709154 14004 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 680c8f54e75a4291bb4d769d94e21552 consensus state has no leader
W20251211 08:57:58.715958 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 238823us
W20251211 08:57:58.716526 14382 leader_election.cc:343] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:57:58.717952 14382 leader_election.cc:343] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:58.718276 14382 leader_election.cc:304] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:58.719081 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed0e90efe651499c929b1fb992712b96" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:58.720607 14722 tablet_bootstrap.cc:492] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:58.723932 14382 leader_election.cc:304] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:58.726531 14268 tablet_service.cc:1505] Processing CreateTablet for tablet 10d8b7117acd41a9a6cc356ed51bb6e8 (DEFAULT_TABLE table=default.Table_a4dd5045e2db4928b29f19e423ba3188 [id=92f301dfe9bf45efa44c96ceea930854]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:58.727548 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d8b7117acd41a9a6cc356ed51bb6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:58.730466 14722 tablet_bootstrap.cc:654] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:58.742933 14655 heartbeater.cc:344] Connected to a master server at 127.13.146.62:39015
I20251211 08:57:58.747233 16125 raft_consensus.cc:2749] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:58.752223 14714 raft_consensus.cc:359] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.752867 14714 raft_consensus.cc:385] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:58.753113 14714 raft_consensus.cc:740] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:58.753167 15950 raft_consensus.cc:493] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:58.753618 15950 raft_consensus.cc:515] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.753630 14714 consensus_queue.cc:260] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.755410 15950 leader_election.cc:290] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:58.755730 16122 raft_consensus.cc:2749] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:58.757032 14714 ts_tablet_manager.cc:1434] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.051s user 0.005s sys 0.001s
I20251211 08:57:58.759001 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_7ee755c24982464a9d02be456caa647e" } new_table_name: "default.Table_3528b4ed9d9742dda51c5fd712140bd7" modify_external_catalogs: true
I20251211 08:57:58.764516 14012 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_485bb3815ac5463185922d1db5a7da52" } modify_external_catalogs: true
I20251211 08:57:58.765079 14012 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_485bb3815ac5463185922d1db5a7da52" } modify_external_catalogs: true
I20251211 08:57:58.771287 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_dbdc2eaf4309442888ea49a832da3967" } modify_external_catalogs: true
I20251211 08:57:58.771756 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_dbdc2eaf4309442888ea49a832da3967" } modify_external_catalogs: true
W20251211 08:57:58.772495 14542 leader_election.cc:343] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:58.786651 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_2aa82bb39d744859b7dc4e25a5c495f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:58.788555 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aa82bb39d744859b7dc4e25a5c495f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.791406 15941 tablet_replica.cc:333] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:58.792181 15941 raft_consensus.cc:2243] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:58.793820 16126 raft_consensus.cc:2804] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:58.793923 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c112dfcc301648cda6f0f038b1f2af83" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:57:58.795274 14542 consensus_peers.cc:597] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a1b011cb7c647f79bb16ce1c92bcff5. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:57:58.795603 14231 leader_election.cc:343] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:58.797760 15941 raft_consensus.cc:2272] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:58.799602 15941 ts_tablet_manager.cc:1916] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:58.800398 16126 raft_consensus.cc:697] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:58.803838 14010 master_service.cc:954] ReplaceTablet: received request to replace tablet c55408f39a414e2caf1db48e6f29eca7 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:58.801317 16126 consensus_queue.cc:237] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.811223 14905 ts_tablet_manager.cc:1929] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251211 08:57:58.811606 14905 log.cc:1199] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/da57b72207414512b9744b283bc869fe
I20251211 08:57:58.812290 14905 ts_tablet_manager.cc:1950] T da57b72207414512b9744b283bc869fe P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:58.813982 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 409) took 9705 ms. Trace:
I20251211 08:57:58.826478 14722 tablet_bootstrap.cc:492] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:58.829979 14722 ts_tablet_manager.cc:1403] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.110s user 0.014s sys 0.002s
I20251211 08:57:58.843253 14010 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet c55408f39a414e2caf1db48e6f29eca7
I20251211 08:57:58.844388 14010 catalog_manager.cc:6724] ReplaceTablet: tablet c55408f39a414e2caf1db48e6f29eca7 deleted and replaced by tablet 52ae270f69724ef4b11f8831906add57
I20251211 08:57:58.847146 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_cd8592260266488598378cde91871ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:58.848400 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd8592260266488598378cde91871ad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.852644 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_17ce34fa6fde45f1950f008acbfdf90c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:58.855741 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17ce34fa6fde45f1950f008acbfdf90c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:58.845881 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet c55408f39a414e2caf1db48e6f29eca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 52ae270f69724ef4b11f8831906add57) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:58.838735 14722 raft_consensus.cc:359] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.861753 14722 raft_consensus.cc:385] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:58.862049 14722 raft_consensus.cc:740] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:58.862689 14722 consensus_queue.cc:260] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.868708 14012 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 8e6f28ebc5524c8a9dad7e9458b98158
I20251211 08:57:58.870626 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 2bf678f74a12464f9361e379e8af0522
I20251211 08:57:58.872684 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet 2bf678f74a12464f9361e379e8af0522 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:58 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:58.878295 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 8fa08edc992c4e84b3defd99472506ba (DEFAULT_TABLE table=default.Table_39b677204e454017b7cc28e910fa9b30 [id=95c0111cd3204a39a0f922c40ac177cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:58.864712 14722 ts_tablet_manager.cc:1434] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.033s user 0.003s sys 0.001s
I20251211 08:57:58.879344 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fa08edc992c4e84b3defd99472506ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:58.879637 14722 tablet_bootstrap.cc:492] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:58.890352 14722 tablet_bootstrap.cc:654] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:58.891131 14714 tablet_bootstrap.cc:492] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:58.891985 14257 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:46838 (request call id 70) took 1765 ms. Trace:
I20251211 08:57:58.892441 14257 rpcz_store.cc:276] 1211 08:57:57.126531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.126953 (+ 422us) service_pool.cc:225] Handling call
1211 08:57:57.712201 (+585248us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b5400095528. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f146046c1 00007f0f146902d2 00007f0f1529d1e0 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:57:58.255383 (+543182us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54001a28a8. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f146046c1 00007f0f146902d2 00007f0f1529d1e0 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:57:58.891930 (+636547us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":357434496}
I20251211 08:57:58.904515 16131 leader_election.cc:290] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:58.908638 14714 tablet_bootstrap.cc:654] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:58.909397 14382 leader_election.cc:343] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: f2f095b59ecf44ffabaa0ac22212938b
I20251211 08:57:58.913447 14385 leader_election.cc:304] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
W20251211 08:57:58.913955 14385 raft_consensus.cc:2686] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251211 08:57:58.916821 16128 raft_consensus.cc:493] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:58.917253 16128 raft_consensus.cc:515] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.918998 16128 leader_election.cc:290] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:58.921211 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2f095b59ecf44ffabaa0ac22212938b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:58.921779 14413 tablet_service.cc:1505] Processing CreateTablet for tablet 8fa08edc992c4e84b3defd99472506ba (DEFAULT_TABLE table=default.Table_39b677204e454017b7cc28e910fa9b30 [id=95c0111cd3204a39a0f922c40ac177cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:58.922770 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fa08edc992c4e84b3defd99472506ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:58.926923 14414 tablet_service.cc:1552] Processing DeleteTablet for tablet c55408f39a414e2caf1db48e6f29eca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 52ae270f69724ef4b11f8831906add57) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:58.928162 15941 ts_tablet_manager.cc:1929] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:58.928543 15941 log.cc:1199] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/51c3f5eaef784e868e565f81bd51a5e4
I20251211 08:57:58.929142 15941 ts_tablet_manager.cc:1950] T 51c3f5eaef784e868e565f81bd51a5e4 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:58.824998 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1b011cb7c647f79bb16ce1c92bcff5" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.833127 14582 tablet_service.cc:1505] Processing CreateTablet for tablet 10d8b7117acd41a9a6cc356ed51bb6e8 (DEFAULT_TABLE table=default.Table_a4dd5045e2db4928b29f19e423ba3188 [id=92f301dfe9bf45efa44c96ceea930854]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:58.930136 14905 rpcz_store.cc:276] 1211 08:57:49.108742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.119488 (+ 10746us) service_pool.cc:225] Handling call
1211 08:57:58.670437 (+9550949us) ts_tablet_manager.cc:1151] Deleting tablet da57b72207414512b9744b283bc869fe
1211 08:57:58.670482 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.764059 (+ 93577us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.810971 (+ 46912us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.813934 (+ 2963us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9542935}
I20251211 08:57:58.834964 15030 tablet_bootstrap.cc:492] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:58.845043 14576 tablet_service.cc:1552] Processing DeleteTablet for tablet c55408f39a414e2caf1db48e6f29eca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 52ae270f69724ef4b11f8831906add57) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:58.851963 16141 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:42846 (request call id 479) took 2103 ms. Trace:
I20251211 08:57:58.931231 15030 ts_tablet_manager.cc:1403] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.267s user 0.009s sys 0.006s
I20251211 08:57:58.931932 14414 tablet_service.cc:1552] Processing DeleteTablet for tablet 2bf678f74a12464f9361e379e8af0522 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:58 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:58.933462 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d8b7117acd41a9a6cc356ed51bb6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:58.931442 16141 rpcz_store.cc:276] 1211 08:57:56.748934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:56.749026 (+ 92us) service_pool.cc:225] Handling call
1211 08:57:58.851908 (+2102882us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:56.749886 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1211 08:57:56.750167 (+ 281us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1211 08:57:56.750191 (+ 24us) alter_schema_op.cc:66] Acquired schema lock
1211 08:57:56.750253 (+ 62us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1211 08:57:56.750354 (+ 101us) alter_schema_op.cc:128] START. Timestamp: 1765443476750308
1211 08:57:56.750374 (+ 20us) op_driver.cc:348] REPLICATION: starting
1211 08:57:56.750785 (+ 411us) log.cc:844] Serialized 173 byte log entry
1211 08:57:58.491279 (+1740494us) op_driver.cc:464] REPLICATION: finished
1211 08:57:58.580617 (+ 89338us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1211 08:57:58.623341 (+ 42724us) log.cc:844] Serialized 5 byte log entry
1211 08:57:58.831523 (+208182us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.851689 (+ 20166us) log.cc:844] Serialized 14 byte log entry
1211 08:57:58.851793 (+ 104us) alter_schema_op.cc:73] Released schema lock
1211 08:57:58.851820 (+ 27us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":89168,"prepare.queue_time_us":151,"prepare.run_cpu_time_us":7026,"prepare.run_wall_time_us":13449,"replication_time_us":1740809,"thread_start_us":288469,"threads_started":3,"wal-append.queue_time_us":189231,"wal-append.run_cpu_time_us":1181,"wal-append.run_wall_time_us":1001223}]]}
I20251211 08:57:58.933298 15030 raft_consensus.cc:359] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.933898 15030 raft_consensus.cc:385] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:58.855890 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c112dfcc301648cda6f0f038b1f2af83" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.934123 15030 raft_consensus.cc:740] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:58.875186 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 2bf678f74a12464f9361e379e8af0522 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:58 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:58.920190 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fdc020f21aa4f35880a435d7cfb605d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:58.934670 15030 consensus_queue.cc:260] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:57:58.935782 14382 leader_election.cc:343] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:58.937484 15030 ts_tablet_manager.cc:1434] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251211 08:57:58.938324 15030 tablet_bootstrap.cc:492] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:58.939524 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet da57b72207414512b9744b283bc869fe (table default.Table_2f325f51dd9f433f98cfef7169cdbe0c [id=2ff95cefc4a545398f979e2bd0321f5a]) successfully deleted
I20251211 08:57:58.939728 14905 tablet_replica.cc:333] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:58.940650 14905 raft_consensus.cc:2243] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:58.941339 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_0597df289bfd46ad9628bfdf81aecb3d" } new_table_name: "default.Table_781755712e1c4e5eb0414cdfd1117e54" modify_external_catalogs: true
W20251211 08:57:58.936363 14385 leader_election.cc:343] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:58.942764 14385 leader_election.cc:304] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:58.943393 14905 raft_consensus.cc:2272] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:58.945170 14288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fdc020f21aa4f35880a435d7cfb605d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:58.945742 14722 tablet_bootstrap.cc:492] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:58.946166 14722 ts_tablet_manager.cc:1403] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.001s
I20251211 08:57:58.948897 14722 raft_consensus.cc:359] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.949476 14722 raft_consensus.cc:385] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:58.949718 14722 raft_consensus.cc:740] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:58.949726 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:57:58.950349 14722 consensus_queue.cc:260] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.952422 14722 ts_tablet_manager.cc:1434] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251211 08:57:58.953115 14722 tablet_bootstrap.cc:492] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:58.953634 14714 tablet_bootstrap.cc:492] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:58.954003 14714 ts_tablet_manager.cc:1403] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.000s
I20251211 08:57:58.954923 16128 raft_consensus.cc:2749] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251211 08:57:58.956130 14714 raft_consensus.cc:359] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.956682 14714 raft_consensus.cc:385] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:58.956943 14714 raft_consensus.cc:740] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
W20251211 08:57:58.957379 14385 leader_election.cc:343] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:58.957731 14385 leader_election.cc:304] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:58.958309 16128 raft_consensus.cc:2749] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:58.961915 15950 consensus_queue.cc:1048] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:58.968283 14905 ts_tablet_manager.cc:1916] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:58.968559 14722 tablet_bootstrap.cc:654] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:58.985739 14722 tablet_bootstrap.cc:492] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:58.986142 14722 ts_tablet_manager.cc:1403] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.000s
I20251211 08:57:58.856159 14230 leader_election.cc:304] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:58.988077 16142 raft_consensus.cc:3060] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:58.988929 14722 raft_consensus.cc:359] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.989499 15030 tablet_bootstrap.cc:654] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:58.989488 14722 raft_consensus.cc:385] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:58.989727 14722 raft_consensus.cc:740] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:58.990218 14722 consensus_queue.cc:260] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:58.991133 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 532) took 1665 ms. Trace:
I20251211 08:57:58.991438 15941 rpcz_store.cc:276] 1211 08:57:57.326118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.326228 (+ 110us) service_pool.cc:225] Handling call
1211 08:57:58.705253 (+1379025us) ts_tablet_manager.cc:1151] Deleting tablet 51c3f5eaef784e868e565f81bd51a5e4
1211 08:57:58.791273 (+ 86020us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.894176 (+102903us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.927969 (+ 33793us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.991080 (+ 63111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5129,"tablet-delete.queue_time_us":1378599}
I20251211 08:57:58.992501 15941 tablet_replica.cc:333] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:58.992499 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 51c3f5eaef784e868e565f81bd51a5e4 (table default.Table_7343d67071c54141af5a57615f82221e [id=fe5786801bab48888480739b8ff266e8]) successfully deleted
I20251211 08:57:58.993716 14722 ts_tablet_manager.cc:1434] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.007s user 0.001s sys 0.007s
I20251211 08:57:58.994484 14722 tablet_bootstrap.cc:492] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:58.995357 14904 ts_tablet_manager.cc:1929] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:59.002456 14714 consensus_queue.cc:260] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.003968 14714 ts_tablet_manager.cc:1434] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.050s user 0.005s sys 0.000s
I20251211 08:57:59.014717 14582 tablet_service.cc:1505] Processing CreateTablet for tablet 8fa08edc992c4e84b3defd99472506ba (DEFAULT_TABLE table=default.Table_39b677204e454017b7cc28e910fa9b30 [id=95c0111cd3204a39a0f922c40ac177cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.018163 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fa08edc992c4e84b3defd99472506ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.020987 14414 tablet_service.cc:1505] Processing CreateTablet for tablet a367e84c55414309bf73796eb3cce392 (DEFAULT_TABLE table=default.Table_a6d23dddea0d46189b02672fbc0c3d04 [id=adcfc5f77ed041f39b8faf6d6ebf439c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.021977 14414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a367e84c55414309bf73796eb3cce392. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.022979 14904 log.cc:1199] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/c6d0fef50524476dbd47fc70e4bda319
W20251211 08:57:59.023757 13999 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 680c8f54e75a4291bb4d769d94e21552 (table default.Table_032686fd837d4292bd6c95749e67b5de [id=387069b529984870961fe2e545fbb221], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 680c8f54e75a4291bb4d769d94e21552 consensus state has no leader
W20251211 08:57:59.024261 14004 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d55c67cd21d141a593e3889032490dbc consensus state has no leader
I20251211 08:57:59.009394 14722 tablet_bootstrap.cc:654] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
W20251211 08:57:59.014501 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 158483us
W20251211 08:57:59.027004 14230 leader_election.cc:343] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 00e5c4bf71b44abca66b0e94995c533e
I20251211 08:57:59.031062 14714 tablet_bootstrap.cc:492] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:59.035075 14905 ts_tablet_manager.cc:1929] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:59.035375 14905 log.cc:1199] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/e9905227c0a844d199362a442e7d630c
I20251211 08:57:59.035992 14905 ts_tablet_manager.cc:1950] T e9905227c0a844d199362a442e7d630c P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:57:59.042327 14230 leader_election.cc:304] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:59.044956 14714 tablet_bootstrap.cc:654] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:59.041005 14904 ts_tablet_manager.cc:1950] T c6d0fef50524476dbd47fc70e4bda319 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:57:59.040352 15030 tablet_bootstrap.cc:492] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:59.047115 15030 ts_tablet_manager.cc:1403] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.109s user 0.005s sys 0.004s
I20251211 08:57:59.053686 15030 raft_consensus.cc:359] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.057551 15030 raft_consensus.cc:385] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:59.057454 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_40b8667dac894419bfd51474ff42c299"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:59.057893 15030 raft_consensus.cc:740] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
W20251211 08:57:59.059638 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40b8667dac894419bfd51474ff42c299 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:59.059808 16131 raft_consensus.cc:493] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:59.060318 16131 raft_consensus.cc:515] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.062063 16131 leader_election.cc:290] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:59.059293 15030 consensus_queue.cc:260] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.068702 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 407) took 9960 ms. Trace:
I20251211 08:57:59.069531 14576 tablet_service.cc:1505] Processing CreateTablet for tablet a367e84c55414309bf73796eb3cce392 (DEFAULT_TABLE table=default.Table_a6d23dddea0d46189b02672fbc0c3d04 [id=adcfc5f77ed041f39b8faf6d6ebf439c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.069655 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570c7f08c46e4273bd38c32593dcf304" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:59.072364 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a367e84c55414309bf73796eb3cce392. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.073190 14230 leader_election.cc:304] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:59.072917 14905 rpcz_store.cc:276] 1211 08:57:49.108228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.119086 (+ 10858us) service_pool.cc:225] Handling call
1211 08:57:58.931040 (+9811954us) ts_tablet_manager.cc:1151] Deleting tablet e9905227c0a844d199362a442e7d630c
1211 08:57:58.939629 (+ 8589us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.998619 (+ 58990us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:59.034881 (+ 36262us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:59.068654 (+ 33773us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9803262}
I20251211 08:57:59.074020 14010 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_5e20e9bb13c4482b8b1a3dca8ef768c8" } modify_external_catalogs: true
I20251211 08:57:59.074434 14010 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_5e20e9bb13c4482b8b1a3dca8ef768c8" } modify_external_catalogs: true
I20251211 08:57:59.077075 15941 raft_consensus.cc:2243] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:59.077450 15941 pending_rounds.cc:70] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:59.070325 15030 ts_tablet_manager.cc:1434] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.023s user 0.003s sys 0.001s
I20251211 08:57:59.077622 15941 pending_rounds.cc:77] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256485265342464 op_type: NO_OP noop_request { }
I20251211 08:57:59.077878 15941 raft_consensus.cc:2889] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:59.078153 15941 raft_consensus.cc:2272] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:58.950666 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_7ee755c24982464a9d02be456caa647e [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 2 got 0
I20251211 08:57:59.079384 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:57:59.081954 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 0ed2fb673bdb4e3da86dca9ff22c6c53 (DEFAULT_TABLE table=default.Table_754c328a447a49b8ab7cecad8c5596dd [id=0db355ebf75f4d11a0566082b1e39683]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.087225 15941 ts_tablet_manager.cc:1916] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:59.087414 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet e9905227c0a844d199362a442e7d630c (table default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b [id=45f67c67257e428f8ed2f16898ad5fe6]) successfully deleted
I20251211 08:57:59.091846 14011 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_17ce34fa6fde45f1950f008acbfdf90c" } modify_external_catalogs: true
I20251211 08:57:59.092361 14011 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_17ce34fa6fde45f1950f008acbfdf90c" } modify_external_catalogs: true
I20251211 08:57:59.093159 14414 tablet_service.cc:1505] Processing CreateTablet for tablet 8dbc6969bbee41da894cac247f1b21f1 (DEFAULT_TABLE table=default.Table_3a6c01d8376b45348eb3e5262f0d5d78 [id=0d901b7468d5471a9c32d2c263218f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.094127 14414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dbc6969bbee41da894cac247f1b21f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.102991 14010 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 3fe773ad17f342f1b9043b0c06004cfb
I20251211 08:57:59.104064 14413 tablet_service.cc:1505] Processing CreateTablet for tablet 0ed2fb673bdb4e3da86dca9ff22c6c53 (DEFAULT_TABLE table=default.Table_754c328a447a49b8ab7cecad8c5596dd [id=0db355ebf75f4d11a0566082b1e39683]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.107120 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ed2fb673bdb4e3da86dca9ff22c6c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.106153 14599 raft_consensus.cc:2468] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:59.111635 14382 leader_election.cc:304] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:59.112313 16126 raft_consensus.cc:2804] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:59.112571 16126 raft_consensus.cc:493] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:59.112803 16126 raft_consensus.cc:3060] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:59.114097 16092 consensus_queue.cc:1048] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:59.119589 15030 tablet_bootstrap.cc:492] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
W20251211 08:57:59.122925 14649 debug-util.cc:398] Leaking SignalData structure 0x7b08000cc500 after lost signal to thread 14508
W20251211 08:57:59.124073 14649 debug-util.cc:398] Leaking SignalData structure 0x7b0800169f00 after lost signal to thread 14652
I20251211 08:57:59.129155 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ed2fb673bdb4e3da86dca9ff22c6c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.132475 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_ac7c79920e5d4e0fbb1298e4533e545f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:59.133836 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac7c79920e5d4e0fbb1298e4533e545f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:59.135172 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_6229702ff48d405ab70d847e8418262a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:59.136458 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6229702ff48d405ab70d847e8418262a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:59.138041 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_c58caf908ea44ee18089745fb0a5c6da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:59.139246 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c58caf908ea44ee18089745fb0a5c6da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:59.139199 16126 raft_consensus.cc:515] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.141799 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570c7f08c46e4273bd38c32593dcf304" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:59.142372 14599 raft_consensus.cc:3060] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:59.149842 14011 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 136baab0fc154f23afdd6650498cef06
I20251211 08:57:59.151688 15030 tablet_bootstrap.cc:654] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:59.163075 14576 tablet_service.cc:1505] Processing CreateTablet for tablet 8dbc6969bbee41da894cac247f1b21f1 (DEFAULT_TABLE table=default.Table_3a6c01d8376b45348eb3e5262f0d5d78 [id=0d901b7468d5471a9c32d2c263218f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.163936 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dbc6969bbee41da894cac247f1b21f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.165688 14599 raft_consensus.cc:2468] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:59.167650 14419 tablet_service.cc:1505] Processing CreateTablet for tablet 48405b8bd9b04c86acbe4376cd88838a (DEFAULT_TABLE table=default.Table_2f6e0f1a6bbf4ed1a761612d13f150af [id=ac53bf72d1ba47a48cdeffbd665398dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.168727 14382 leader_election.cc:304] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:59.169011 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48405b8bd9b04c86acbe4376cd88838a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.169414 16128 raft_consensus.cc:2804] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:59.169816 16128 raft_consensus.cc:697] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:59.170615 16128 consensus_queue.cc:237] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.174901 16126 leader_election.cc:290] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:59.180721 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 48405b8bd9b04c86acbe4376cd88838a (DEFAULT_TABLE table=default.Table_2f6e0f1a6bbf4ed1a761612d13f150af [id=ac53bf72d1ba47a48cdeffbd665398dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.183537 14714 tablet_bootstrap.cc:492] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:59.184043 14714 ts_tablet_manager.cc:1403] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.153s user 0.009s sys 0.005s
I20251211 08:57:59.196568 14414 tablet_service.cc:1505] Processing CreateTablet for tablet 3aed0129e9164a779b73994b8b37426c (DEFAULT_TABLE table=default.Table_dbc6fca2f55b4ce9b55985aec36a4a59 [id=61e04d100239441c8c9f4d085b52cd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.197465 14905 tablet_replica.cc:333] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:57:59.198858 14230 raft_consensus.cc:2686] T 8e05628011634149b17d298c54074d00 P 31404d5f58e548519239569b150d975b: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251211 08:57:59.199335 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 126250us
I20251211 08:57:59.186415 14714 raft_consensus.cc:359] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.199952 14230 leader_election.cc:304] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:59.199047 14414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aed0129e9164a779b73994b8b37426c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:57:59.203040 14230 raft_consensus.cc:2686] T 07aef1eea7c04d9cbf35c82e45df7a16 P 31404d5f58e548519239569b150d975b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251211 08:57:59.198688 16096 raft_consensus.cc:2749] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:59.205418 14905 raft_consensus.cc:2243] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Raft consensus shutting down.
I20251211 08:57:59.206142 14905 pending_rounds.cc:70] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:57:59.206385 14905 pending_rounds.cc:77] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 2 index: 3 } timestamp: 7231256472511721472 op_type: NO_OP noop_request { }
I20251211 08:57:59.206705 14905 raft_consensus.cc:2889] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:57:59.207863 14905 raft_consensus.cc:2272] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:59.209708 14905 ts_tablet_manager.cc:1916] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:59.200094 14714 raft_consensus.cc:385] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:59.212397 14714 raft_consensus.cc:740] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:59.213312 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48405b8bd9b04c86acbe4376cd88838a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.227036 14714 consensus_queue.cc:260] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.227664 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_eb245e8feaab450784ce26f23c5575f8" } new_table_name: "default.Table_1a44912eaeed406d8b0d00b2a210841a" modify_external_catalogs: true
I20251211 08:57:59.229763 14582 tablet_service.cc:1505] Processing CreateTablet for tablet 3aed0129e9164a779b73994b8b37426c (DEFAULT_TABLE table=default.Table_dbc6fca2f55b4ce9b55985aec36a4a59 [id=61e04d100239441c8c9f4d085b52cd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.232632 15941 ts_tablet_manager.cc:1929] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:59.236258 14579 tablet_service.cc:1505] Processing CreateTablet for tablet 0dfa9b3705a14774bd690d3a2a233d48 (DEFAULT_TABLE table=default.Table_eb245e8feaab450784ce26f23c5575f8 [id=c7456283b67e4df7b35bc7f9c832cc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.239567 14009 master_service.cc:954] ReplaceTablet: received request to replace tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 from {username='slave'} at 127.0.0.1:55614
I20251211 08:57:59.245055 15030 tablet_bootstrap.cc:492] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:57:59.245529 15030 ts_tablet_manager.cc:1403] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.126s user 0.010s sys 0.001s
I20251211 08:57:59.247619 15030 raft_consensus.cc:359] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.248322 15030 raft_consensus.cc:385] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251211 08:57:59.253366 14004 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0dfa9b3705a14774bd690d3a2a233d48 consensus state has no leader
I20251211 08:57:59.232062 14714 ts_tablet_manager.cc:1434] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.048s user 0.004s sys 0.000s
I20251211 08:57:59.238903 14413 tablet_service.cc:1505] Processing CreateTablet for tablet b3daeab5843247ee8ab3b5a3e5207fa8 (DEFAULT_TABLE table=default.Table_4c5cbb3bc18748dd81a8641009991654 [id=473aa59c86a542c79f468a6f540c3347]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.260540 15941 log.cc:1199] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/f3403116517f45c59f87fa5bda0b4071
I20251211 08:57:59.261240 15941 ts_tablet_manager.cc:1950] T f3403116517f45c59f87fa5bda0b4071 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:59.270265 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 62c8dcb36e694cf896bd1283db63b4cc (DEFAULT_TABLE table=default.Table_5b713a62983c42f4927bf2e96686ced0 [id=74ec01634f3a4dee9a90892189ff6462]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.249567 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dfa9b3705a14774bd690d3a2a233d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.270596 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (table default.Table_321194b0f71d457e81e14c810650b486 [id=f74e074787144a21ad31d4d89cc509c9]). Expected version 1 got 0
I20251211 08:57:59.251277 14585 tablet_service.cc:1505] Processing CreateTablet for tablet b3daeab5843247ee8ab3b5a3e5207fa8 (DEFAULT_TABLE table=default.Table_4c5cbb3bc18748dd81a8641009991654 [id=473aa59c86a542c79f468a6f540c3347]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.275384 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3daeab5843247ee8ab3b5a3e5207fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.275132 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:57:59.277894 16137 tablet.cc:1722] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:59.279726 14714 tablet_bootstrap.cc:492] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:59.278903 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8
I20251211 08:57:59.284190 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 with delete_type TABLET_DATA_DELETED (replaced by tablet c9975bfc37b14461be5e3066918dac0c) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:59.287007 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aed0129e9164a779b73994b8b37426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.288940 14009 catalog_manager.cc:6724] ReplaceTablet: tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 deleted and replaced by tablet c9975bfc37b14461be5e3066918dac0c
I20251211 08:57:59.289963 14575 tablet_service.cc:1552] Processing DeleteTablet for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 with delete_type TABLET_DATA_DELETED (replaced by tablet c9975bfc37b14461be5e3066918dac0c) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:59.290724 14424 tablet_service.cc:1505] Processing CreateTablet for tablet fb87ed1cb9ac418eaff7676291396ca5 (DEFAULT_TABLE table=default.Table_da9fb81b38a14423b6191db9f7a7c3df [id=e10eaa167f0a46c39674a5e2212f2226]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.291292 14714 tablet_bootstrap.cc:654] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:59.291500 16141 tablet.cc:1722] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:59.294518 14419 tablet_service.cc:1505] Processing CreateTablet for tablet 0dfa9b3705a14774bd690d3a2a233d48 (DEFAULT_TABLE table=default.Table_eb245e8feaab450784ce26f23c5575f8 [id=c7456283b67e4df7b35bc7f9c832cc0c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:57:59.295526 14541 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251211 08:57:59.299459 14268 tablet_service.cc:1505] Processing CreateTablet for tablet a367e84c55414309bf73796eb3cce392 (DEFAULT_TABLE table=default.Table_a6d23dddea0d46189b02672fbc0c3d04 [id=adcfc5f77ed041f39b8faf6d6ebf439c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.300259 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62c8dcb36e694cf896bd1283db63b4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.301445 15030 raft_consensus.cc:740] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:57:59.302177 15030 consensus_queue.cc:260] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.302716 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 0ed2fb673bdb4e3da86dca9ff22c6c53 (DEFAULT_TABLE table=default.Table_754c328a447a49b8ab7cecad8c5596dd [id=0db355ebf75f4d11a0566082b1e39683]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.304446 15030 ts_tablet_manager.cc:1434] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.059s user 0.004s sys 0.000s
I20251211 08:57:59.309669 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_91bdd3847f834c4b8500113d994514a5" } modify_external_catalogs: true
I20251211 08:57:59.310206 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_91bdd3847f834c4b8500113d994514a5" } modify_external_catalogs: true
I20251211 08:57:59.295027 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_62edf14ec336499b99d2f34c758edfe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:59.314666 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62edf14ec336499b99d2f34c758edfe4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:59.317993 14586 tablet_service.cc:1505] Processing CreateTablet for tablet fb87ed1cb9ac418eaff7676291396ca5 (DEFAULT_TABLE table=default.Table_da9fb81b38a14423b6191db9f7a7c3df [id=e10eaa167f0a46c39674a5e2212f2226]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.319123 16141 tablet.cc:1722] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:59.319833 14415 tablet_service.cc:1505] Processing CreateTablet for tablet 29fb5dcf46324a66ad0cbbb4aed9de9c (DEFAULT_TABLE table=default.Table_c1533583a6fd410f96e4da3f3312ba70 [id=3148a886248d4e598c7187e22b352007]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.323738 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 62c8dcb36e694cf896bd1283db63b4cc (DEFAULT_TABLE table=default.Table_5b713a62983c42f4927bf2e96686ced0 [id=74ec01634f3a4dee9a90892189ff6462]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.290632 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:57:59.327436 15902 raft_consensus.cc:493] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:59.327582 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_7ee755c24982464a9d02be456caa647e [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 2 got 0
I20251211 08:57:59.327834 15902 raft_consensus.cc:515] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.328145 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet f5c5b1aebde94dea81cb37a6a6998070 (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]). Expected version 1 got 0
I20251211 08:57:59.329492 15902 leader_election.cc:290] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:59.331135 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 8dbc6969bbee41da894cac247f1b21f1 (DEFAULT_TABLE table=default.Table_3a6c01d8376b45348eb3e5262f0d5d78 [id=0d901b7468d5471a9c32d2c263218f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.334478 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 48405b8bd9b04c86acbe4376cd88838a (DEFAULT_TABLE table=default.Table_2f6e0f1a6bbf4ed1a761612d13f150af [id=ac53bf72d1ba47a48cdeffbd665398dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.335790 15902 raft_consensus.cc:493] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31404d5f58e548519239569b150d975b)
I20251211 08:57:59.336195 15902 raft_consensus.cc:515] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:59.337188 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet f6fb2096f80a43b5a5a6230a824fceb7
I20251211 08:57:59.338689 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet f6fb2096f80a43b5a5a6230a824fceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:59 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:57:59.339372 14575 tablet_service.cc:1552] Processing DeleteTablet for tablet f6fb2096f80a43b5a5a6230a824fceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:59 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:57:59.340577 14269 tablet_service.cc:1505] Processing CreateTablet for tablet 0dfa9b3705a14774bd690d3a2a233d48 (DEFAULT_TABLE table=default.Table_eb245e8feaab450784ce26f23c5575f8 [id=c7456283b67e4df7b35bc7f9c832cc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.340844 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3daeab5843247ee8ab3b5a3e5207fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.340823 14575 tablet_service.cc:1505] Processing CreateTablet for tablet 29fb5dcf46324a66ad0cbbb4aed9de9c (DEFAULT_TABLE table=default.Table_c1533583a6fd410f96e4da3f3312ba70 [id=3148a886248d4e598c7187e22b352007]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.341184 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 3aed0129e9164a779b73994b8b37426c (DEFAULT_TABLE table=default.Table_dbc6fca2f55b4ce9b55985aec36a4a59 [id=61e04d100239441c8c9f4d085b52cd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.341737 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb87ed1cb9ac418eaff7676291396ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.342041 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570c7f08c46e4273bd38c32593dcf304" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:57:59.342518 14295 raft_consensus.cc:3060] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:59.343189 14415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29fb5dcf46324a66ad0cbbb4aed9de9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.344976 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 1f0f14e797684bea987e5ef104e5d8e0 (DEFAULT_TABLE table=default.Table_13b04a9ecefd46af88c40b23b27dc23b [id=f154e4d1c10b47fdaa102445e1f7c983]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.345781 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f0f14e797684bea987e5ef104e5d8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.338567 15902 leader_election.cc:290] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:57:59.333585 14288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570c7f08c46e4273bd38c32593dcf304" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:59.348489 14542 leader_election.cc:343] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: ccce5a446ac9460c9c9b7de72d875d95
I20251211 08:57:59.348459 16137 tablet.cc:1722] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:57:59.355774 14279 tablet_service.cc:1505] Processing CreateTablet for tablet b3daeab5843247ee8ab3b5a3e5207fa8 (DEFAULT_TABLE table=default.Table_4c5cbb3bc18748dd81a8641009991654 [id=473aa59c86a542c79f468a6f540c3347]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.356819 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111adfd1ac1a42bdafa9416594e7e932" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:57:59.357373 14441 raft_consensus.cc:2468] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:57:59.358224 14542 leader_election.cc:304] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:59.358920 15902 raft_consensus.cc:2804] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:59.359225 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dfa9b3705a14774bd690d3a2a233d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.359256 15902 raft_consensus.cc:493] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:59.359615 15902 raft_consensus.cc:3060] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:59.360122 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_a4ec7037d1b24a1492dc74ae4ea83d06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:57:59.361575 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4ec7037d1b24a1492dc74ae4ea83d06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:59.366061 15902 raft_consensus.cc:515] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.367655 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_d8915549348b457baf42c8b9dfaa98fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:59.360793 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 533) took 1923 ms. Trace:
I20251211 08:57:59.368261 15941 rpcz_store.cc:276] 1211 08:57:57.437571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.437682 (+ 111us) service_pool.cc:225] Handling call
1211 08:57:58.992374 (+1554692us) ts_tablet_manager.cc:1151] Deleting tablet f3403116517f45c59f87fa5bda0b4071
1211 08:57:58.992407 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:59.098458 (+106051us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:59.232433 (+133975us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:59.360727 (+128294us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83810,"tablet-delete.queue_time_us":1554064}
W20251211 08:57:59.368955 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8915549348b457baf42c8b9dfaa98fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:57:59.369165 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet f3403116517f45c59f87fa5bda0b4071 (table default.Table_6109f16f9985461fbb07b1d083bbb166 [id=58f79d91f6e64d7980b96715d3dd25e7]) successfully deleted
I20251211 08:57:59.369658 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 534) took 1881 ms. Trace:
I20251211 08:57:59.369889 15941 rpcz_store.cc:276] 1211 08:57:57.487787 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.487932 (+ 145us) service_pool.cc:225] Handling call
1211 08:57:59.369378 (+1881446us) ts_tablet_manager.cc:1151] Deleting tablet 15595c5a432047de873c2647b78838d6
1211 08:57:59.369429 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:59.369617 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1878107}
I20251211 08:57:59.371045 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 535) took 1830 ms. Trace:
I20251211 08:57:59.371254 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_cbc846a77e59420a9e6d78771903f231"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:57:59.371321 15941 rpcz_store.cc:276] 1211 08:57:57.540233 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.541906 (+ 1673us) service_pool.cc:225] Handling call
1211 08:57:59.370720 (+1828814us) ts_tablet_manager.cc:1151] Deleting tablet 2c0f8d7dd90f4beaae674fe827d37e0b
1211 08:57:59.370771 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:59.371003 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1828256}
W20251211 08:57:59.372491 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbc846a77e59420a9e6d78771903f231 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:57:59.371265 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 15595c5a432047de873c2647b78838d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15595c5a432047de873c2647b78838d6 already in progress: creating tablet
W20251211 08:57:59.375725 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 2c0f8d7dd90f4beaae674fe827d37e0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c0f8d7dd90f4beaae674fe827d37e0b already in progress: creating tablet
I20251211 08:57:59.382301 14414 tablet_service.cc:1505] Processing CreateTablet for tablet 918539839f1d4b5fb90e5d2f810e0d0c (DEFAULT_TABLE table=default.Table_ae53aa073c864155bd414bdee00d29fc [id=42c2a229f9284fdc9b5ca6ce1c233e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.373201 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 1f0f14e797684bea987e5ef104e5d8e0 (DEFAULT_TABLE table=default.Table_13b04a9ecefd46af88c40b23b27dc23b [id=f154e4d1c10b47fdaa102445e1f7c983]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.384060 14414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 918539839f1d4b5fb90e5d2f810e0d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.387073 14581 tablet_service.cc:1505] Processing CreateTablet for tablet 918539839f1d4b5fb90e5d2f810e0d0c (DEFAULT_TABLE table=default.Table_ae53aa073c864155bd414bdee00d29fc [id=42c2a229f9284fdc9b5ca6ce1c233e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.388571 14273 tablet_service.cc:1505] Processing CreateTablet for tablet 62c8dcb36e694cf896bd1283db63b4cc (DEFAULT_TABLE table=default.Table_5b713a62983c42f4927bf2e96686ced0 [id=74ec01634f3a4dee9a90892189ff6462]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.388716 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111adfd1ac1a42bdafa9416594e7e932" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:57:59.388737 14278 tablet_service.cc:1505] Processing CreateTablet for tablet fb87ed1cb9ac418eaff7676291396ca5 (DEFAULT_TABLE table=default.Table_da9fb81b38a14423b6191db9f7a7c3df [id=e10eaa167f0a46c39674a5e2212f2226]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.389184 14441 raft_consensus.cc:3060] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:59.389766 14277 tablet_service.cc:1552] Processing DeleteTablet for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 with delete_type TABLET_DATA_DELETED (replaced by tablet c9975bfc37b14461be5e3066918dac0c) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:59.390033 14272 tablet_service.cc:1552] Processing DeleteTablet for tablet f6fb2096f80a43b5a5a6230a824fceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:59 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:57:59.390851 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 29fb5dcf46324a66ad0cbbb4aed9de9c (DEFAULT_TABLE table=default.Table_c1533583a6fd410f96e4da3f3312ba70 [id=3148a886248d4e598c7187e22b352007]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.392380 14272 tablet_service.cc:1505] Processing CreateTablet for tablet 1f0f14e797684bea987e5ef104e5d8e0 (DEFAULT_TABLE table=default.Table_13b04a9ecefd46af88c40b23b27dc23b [id=f154e4d1c10b47fdaa102445e1f7c983]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.392596 14277 tablet_service.cc:1505] Processing CreateTablet for tablet 918539839f1d4b5fb90e5d2f810e0d0c (DEFAULT_TABLE table=default.Table_ae53aa073c864155bd414bdee00d29fc [id=42c2a229f9284fdc9b5ca6ce1c233e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.393045 14260 tablet_service.cc:1505] Processing CreateTablet for tablet c177fcaa413a4034bc1418d66bd61e46 (DEFAULT_TABLE table=default.Table_94508c2cd6a741c38c98b9f9f44ee938 [id=4e4ddac7980f4041b397cc27a1678d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.389710 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 536) took 1619 ms. Trace:
I20251211 08:57:59.399940 15902 leader_election.cc:290] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:57:59.398766 14418 tablet_service.cc:1505] Processing CreateTablet for tablet c177fcaa413a4034bc1418d66bd61e46 (DEFAULT_TABLE table=default.Table_94508c2cd6a741c38c98b9f9f44ee938 [id=4e4ddac7980f4041b397cc27a1678d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.400416 15941 rpcz_store.cc:276] 1211 08:57:57.770235 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.770365 (+ 130us) service_pool.cc:225] Handling call
1211 08:57:59.389438 (+1619073us) ts_tablet_manager.cc:1151] Deleting tablet eccc727462b447489372da09ea828ac2
1211 08:57:59.389474 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:59.389666 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1618530}
I20251211 08:57:59.400925 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c177fcaa413a4034bc1418d66bd61e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.402410 14577 tablet_service.cc:1505] Processing CreateTablet for tablet c177fcaa413a4034bc1418d66bd61e46 (DEFAULT_TABLE table=default.Table_94508c2cd6a741c38c98b9f9f44ee938 [id=4e4ddac7980f4041b397cc27a1678d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.399560 14441 raft_consensus.cc:2468] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:57:59.403412 14542 leader_election.cc:304] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
W20251211 08:57:59.403496 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet eccc727462b447489372da09ea828ac2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eccc727462b447489372da09ea828ac2 already in progress: creating tablet
I20251211 08:57:59.404197 15902 raft_consensus.cc:2804] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:59.404805 15902 raft_consensus.cc:697] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:57:59.404922 15941 tablet_replica.cc:333] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:59.405696 15941 raft_consensus.cc:2243] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:59.405584 15902 consensus_queue.cc:237] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.407618 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a367e84c55414309bf73796eb3cce392. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.408334 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ed2fb673bdb4e3da86dca9ff22c6c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.408977 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48405b8bd9b04c86acbe4376cd88838a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.409561 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dfa9b3705a14774bd690d3a2a233d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.410254 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3daeab5843247ee8ab3b5a3e5207fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.410208 14574 tablet_service.cc:1505] Processing CreateTablet for tablet d55c67cd21d141a593e3889032490dbc (DEFAULT_TABLE table=default.Table_781755712e1c4e5eb0414cdfd1117e54 [id=a762eb04798c4cffb074ba0e4559ead1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.410939 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aed0129e9164a779b73994b8b37426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.411558 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29fb5dcf46324a66ad0cbbb4aed9de9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.412304 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f0f14e797684bea987e5ef104e5d8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.413035 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 918539839f1d4b5fb90e5d2f810e0d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.413827 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62c8dcb36e694cf896bd1283db63b4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.414522 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb87ed1cb9ac418eaff7676291396ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.416900 14004 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_289d1c8ae20f4dd893b92590045554ad" } modify_external_catalogs: true
W20251211 08:57:59.425830 16070 lock_manager.cc:585] Waited 2 seconds to obtain row lock on key '<redacted>' (key hash 12660783061858645059) tablet 00000000000000000000000000000000 cur holder 0x7b5800045300
I20251211 08:57:59.435324 14263 tablet_service.cc:1505] Processing CreateTablet for tablet d55c67cd21d141a593e3889032490dbc (DEFAULT_TABLE table=default.Table_781755712e1c4e5eb0414cdfd1117e54 [id=a762eb04798c4cffb074ba0e4559ead1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.436285 14263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55c67cd21d141a593e3889032490dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.436951 14905 ts_tablet_manager.cc:1929] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20251211 08:57:59.437345 14905 log.cc:1199] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/e15483f539f24f1ba1b442663d66fd45
I20251211 08:57:59.452821 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dbc6969bbee41da894cac247f1b21f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.456334 15941 raft_consensus.cc:2272] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:59.457962 15941 ts_tablet_manager.cc:1916] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:59.463279 14905 ts_tablet_manager.cc:1950] T e15483f539f24f1ba1b442663d66fd45 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:57:59.467067 14261 consensus_meta.cc:287] T 22c89da854ef4396a1e1f4173020589b P 31404d5f58e548519239569b150d975b: Time spent flushing consensus metadata: real 0.685s user 0.007s sys 0.000s
I20251211 08:57:59.472383 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c177fcaa413a4034bc1418d66bd61e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.460166 14413 tablet_service.cc:1505] Processing CreateTablet for tablet d55c67cd21d141a593e3889032490dbc (DEFAULT_TABLE table=default.Table_781755712e1c4e5eb0414cdfd1117e54 [id=a762eb04798c4cffb074ba0e4559ead1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:57:59.495209 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55c67cd21d141a593e3889032490dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.454452 16126 raft_consensus.cc:493] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:59.497721 16126 raft_consensus.cc:515] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.499511 16126 leader_election.cc:290] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:59.501158 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c0bdd2d0d9f4221bf13812bb7a97025" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:57:59.503024 14382 leader_election.cc:343] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:59.469645 14714 tablet_bootstrap.cc:492] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:59.507089 14714 ts_tablet_manager.cc:1403] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.228s user 0.006s sys 0.004s
I20251211 08:57:59.509477 14714 raft_consensus.cc:359] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.510205 14714 raft_consensus.cc:385] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:59.510494 14714 raft_consensus.cc:740] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:59.511227 14714 consensus_queue.cc:260] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.520140 14714 ts_tablet_manager.cc:1434] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.013s user 0.003s sys 0.001s
I20251211 08:57:59.521029 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c0bdd2d0d9f4221bf13812bb7a97025" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:59.534224 14295 raft_consensus.cc:2468] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
W20251211 08:57:59.522253 14385 leader_election.cc:343] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:59.548959 14385 leader_election.cc:304] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:59.549860 16126 raft_consensus.cc:2749] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:59.551093 14714 tablet_bootstrap.cc:492] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:57:59.571547 16142 consensus_meta.cc:287] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b: Time spent flushing consensus metadata: real 0.583s user 0.006s sys 0.000s
I20251211 08:57:59.585448 16142 raft_consensus.cc:2749] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:59.602007 14714 tablet_bootstrap.cc:654] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:59.616649 14722 tablet_bootstrap.cc:492] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:57:59.617484 14722 ts_tablet_manager.cc:1403] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.623s user 0.023s sys 0.017s
I20251211 08:57:59.622462 15941 ts_tablet_manager.cc:1929] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:57:59.635054 14722 raft_consensus.cc:359] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.635637 14722 raft_consensus.cc:385] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:59.635867 14722 raft_consensus.cc:740] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:57:59.636404 14722 consensus_queue.cc:260] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.638350 14722 ts_tablet_manager.cc:1434] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20251211 08:57:59.456482 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_6c1a460a14a04dfb9cecebd86166d475" } new_table_name: "default.Table_f9599a5390cc46ecba6001364e0f78bb" modify_external_catalogs: true
I20251211 08:57:59.648823 14722 tablet_bootstrap.cc:492] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:57:59.651284 14004 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_289d1c8ae20f4dd893b92590045554ad" } modify_external_catalogs: true
I20251211 08:57:59.652454 14714 tablet_bootstrap.cc:492] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:59.652891 14714 ts_tablet_manager.cc:1403] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.102s user 0.005s sys 0.002s
W20251211 08:57:59.657778 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 453778us
W20251211 08:57:59.658494 14230 leader_election.cc:343] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: dd14de369e084828afb8a924c46c649b
I20251211 08:57:59.658874 15941 log.cc:1199] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/f2f095b59ecf44ffabaa0ac22212938b
I20251211 08:57:59.658975 14230 leader_election.cc:304] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:57:59.659530 15941 ts_tablet_manager.cc:1950] T f2f095b59ecf44ffabaa0ac22212938b P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:59.659704 16096 raft_consensus.cc:2749] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:57:59.660395 14714 raft_consensus.cc:359] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.660827 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 459) took 3866 ms. Trace:
I20251211 08:57:59.661006 14714 raft_consensus.cc:385] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:59.661247 14714 raft_consensus.cc:740] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:59.661281 14904 rpcz_store.cc:276] 1211 08:57:55.793929 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.822272 (+ 28343us) service_pool.cc:225] Handling call
1211 08:57:58.477141 (+2654869us) ts_tablet_manager.cc:1151] Deleting tablet c6d0fef50524476dbd47fc70e4bda319
1211 08:57:58.477188 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:58.686750 (+209562us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:58.995153 (+308403us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:59.660774 (+665621us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":156941,"tablet-delete.queue_time_us":2628320}
I20251211 08:57:59.662635 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet c6d0fef50524476dbd47fc70e4bda319 (table default.Table_7919ce70cfc5400bb4f2515164c6b7a1 [id=3a336d5ad7004619951e901ad9bf39bf]) successfully deleted
I20251211 08:57:59.663921 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 462) took 3869 ms. Trace:
I20251211 08:57:59.664249 14904 rpcz_store.cc:276] 1211 08:57:55.794600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.850272 (+ 55672us) service_pool.cc:225] Handling call
1211 08:57:59.663554 (+3813282us) ts_tablet_manager.cc:1151] Deleting tablet f2dc6cb4efc744e6904728ad120df934
1211 08:57:59.663595 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:59.663859 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3812571}
I20251211 08:57:59.666149 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 537) took 1199 ms. Trace:
I20251211 08:57:59.666460 15941 rpcz_store.cc:276] 1211 08:57:58.467103 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.467322 (+ 219us) service_pool.cc:225] Handling call
1211 08:57:59.404796 (+937474us) ts_tablet_manager.cc:1151] Deleting tablet f2f095b59ecf44ffabaa0ac22212938b
1211 08:57:59.404831 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:59.563044 (+158213us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:59.622236 (+ 59192us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:59.666090 (+ 43854us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22564,"tablet-delete.queue_time_us":936916}
I20251211 08:57:59.667141 16141 tablet.cc:1722] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:59.667425 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet f2f095b59ecf44ffabaa0ac22212938b (table default.Table_fee4bd06cd7e4df08fa4bafdd291a3d4 [id=e754a76685124291a154d2a35952abd3]) successfully deleted
I20251211 08:57:59.667570 14714 consensus_queue.cc:260] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.670226 16137 tablet.cc:1722] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:57:59.706734 14714 ts_tablet_manager.cc:1434] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.053s user 0.004s sys 0.000s
I20251211 08:57:59.707613 14714 tablet_bootstrap.cc:492] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:59.710625 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 541) took 1186 ms. Trace:
I20251211 08:57:59.710988 15941 rpcz_store.cc:276] 1211 08:57:58.523899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.524252 (+ 353us) service_pool.cc:225] Handling call
1211 08:57:59.710314 (+1186062us) ts_tablet_manager.cc:1151] Deleting tablet 020230ff08ed42179a28efbb21613a0f
1211 08:57:59.710352 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:59.710561 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007259}
W20251211 08:57:59.718597 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 020230ff08ed42179a28efbb21613a0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 020230ff08ed42179a28efbb21613a0f already in progress: creating tablet
I20251211 08:57:59.730723 15941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 542) took 1167 ms. Trace:
I20251211 08:57:59.731068 15941 rpcz_store.cc:276] 1211 08:57:58.563184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.563325 (+ 141us) service_pool.cc:225] Handling call
1211 08:57:59.730408 (+1167083us) ts_tablet_manager.cc:1151] Deleting tablet 3aed0129e9164a779b73994b8b37426c
1211 08:57:59.730448 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:59.730673 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026348}
W20251211 08:57:59.737632 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 3aed0129e9164a779b73994b8b37426c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aed0129e9164a779b73994b8b37426c already in progress: creating tablet
I20251211 08:57:59.740887 16126 raft_consensus.cc:493] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:59.741284 16126 raft_consensus.cc:515] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.743000 16126 leader_election.cc:290] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:59.744428 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed0e90efe651499c929b1fb992712b96" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:59.744731 14007 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e3591ad0b25d41499ce64dc2ea8ff7b9" } modify_external_catalogs: true
I20251211 08:57:59.745004 14599 raft_consensus.cc:2468] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:57:59.745260 14007 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_e3591ad0b25d41499ce64dc2ea8ff7b9" } modify_external_catalogs: true
I20251211 08:57:59.746392 14382 leader_election.cc:304] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:59.747714 16126 raft_consensus.cc:2804] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:57:59.748109 16126 raft_consensus.cc:493] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:57:59.748502 16126 raft_consensus.cc:3060] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:59.751034 14497 heartbeater.cc:344] Connected to a master server at 127.13.146.62:39015
I20251211 08:57:59.754987 16128 consensus_queue.cc:1048] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:57:59.762319 16126 raft_consensus.cc:515] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.789032 16128 raft_consensus.cc:493] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:59.789429 16128 raft_consensus.cc:515] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.792209 16128 leader_election.cc:290] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:59.793814 14714 tablet_bootstrap.cc:654] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:59.794873 16126 leader_election.cc:290] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:59.805526 15941 tablet_replica.cc:333] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:59.806377 15941 raft_consensus.cc:2243] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:59.806917 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c112dfcc301648cda6f0f038b1f2af83" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:59.807284 15941 raft_consensus.cc:2272] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:59.809227 15941 ts_tablet_manager.cc:1916] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:59.809098 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed0e90efe651499c929b1fb992712b96" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:57:59.809705 14599 raft_consensus.cc:3060] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:59.813715 15030 tablet_bootstrap.cc:492] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:57:59.817183 14599 raft_consensus.cc:2468] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:57:59.821290 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb87ed1cb9ac418eaff7676291396ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.821527 16184 raft_consensus.cc:493] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:57:59.822094 16184 raft_consensus.cc:515] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.824975 16184 leader_election.cc:290] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:57:59.825621 14714 tablet_bootstrap.cc:492] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
W20251211 08:57:59.825623 14382 leader_election.cc:343] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:59.826097 14714 ts_tablet_manager.cc:1403] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.119s user 0.000s sys 0.010s
I20251211 08:57:59.826303 14382 leader_election.cc:304] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:57:59.827194 16184 raft_consensus.cc:2804] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:57:59.827716 16184 raft_consensus.cc:697] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:57:59.828971 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62c8dcb36e694cf896bd1283db63b4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.828758 16184 consensus_queue.cc:237] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.829250 14714 raft_consensus.cc:359] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:59.829909 14714 raft_consensus.cc:385] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:59.830183 14714 raft_consensus.cc:740] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:57:59.830909 16181 consensus_queue.cc:1048] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:57:59.832221 14544 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251211 08:57:59.838518 15941 ts_tablet_manager.cc:1929] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:59.838757 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b2445f66be54d3fbdc3be964b9a9132" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:57:59.843511 14575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29fb5dcf46324a66ad0cbbb4aed9de9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.848493 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f0f14e797684bea987e5ef104e5d8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.851194 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 918539839f1d4b5fb90e5d2f810e0d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.853456 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 412) took 10663 ms. Trace:
I20251211 08:57:59.853739 14905 rpcz_store.cc:276] 1211 08:57:49.190424 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.190532 (+ 108us) service_pool.cc:225] Handling call
1211 08:57:59.077626 (+9887094us) ts_tablet_manager.cc:1151] Deleting tablet e15483f539f24f1ba1b442663d66fd45
1211 08:57:59.197350 (+119724us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:59.383270 (+185920us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:59.436702 (+ 53432us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:59.853407 (+416705us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9886619}
I20251211 08:57:59.854626 15030 tablet_bootstrap.cc:654] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:59.854712 14905 tablet_replica.cc:333] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:59.856192 14905 raft_consensus.cc:2243] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:57:59.856695 14905 raft_consensus.cc:2272] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:59.855044 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c177fcaa413a4034bc1418d66bd61e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.860977 15941 log.cc:1199] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/c55408f39a414e2caf1db48e6f29eca7
I20251211 08:57:59.861689 15941 ts_tablet_manager.cc:1950] T c55408f39a414e2caf1db48e6f29eca7 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:57:59.860205 14714 consensus_queue.cc:260] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:59.862975 14574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55c67cd21d141a593e3889032490dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:57:59.863551 14714 ts_tablet_manager.cc:1434] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.037s user 0.000s sys 0.007s
I20251211 08:57:59.864369 14714 tablet_bootstrap.cc:492] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:57:59.870280 14714 tablet_bootstrap.cc:654] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:59.875607 14905 ts_tablet_manager.cc:1916] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:57:59.887568 14382 leader_election.cc:343] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:59.713461 14722 tablet_bootstrap.cc:654] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:57:59.764147 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed0e90efe651499c929b1fb992712b96" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:57:59.783270 14231 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.009s sys 0.001s
W20251211 08:57:59.895797 14231 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.692s user 0.010s sys 0.001s
W20251211 08:57:59.787060 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 128660us
I20251211 08:57:59.792721 14286 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c112dfcc301648cda6f0f038b1f2af83" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:57:59.969247 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet e15483f539f24f1ba1b442663d66fd45 (table default.Table_49a263a32c9940a5b61d8a430ebb61a7 [id=69dd3ef114034ac7a3d60b8a0fde70b6]) successfully deleted
I20251211 08:57:59.969856 14905 ts_tablet_manager.cc:1929] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:57:59.970598 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet c55408f39a414e2caf1db48e6f29eca7 (table default.Table_2a5d1e8d612c4fc5954ea3efc00eb00b [id=ad68b61e866c40f28e5dbd143ccfcb55]) successfully deleted
I20251211 08:57:59.975353 14905 log.cc:1199] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/080d1700df4a4df09acda65c6d021d96
W20251211 08:57:59.975205 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 2bf678f74a12464f9361e379e8af0522 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bf678f74a12464f9361e379e8af0522 already in progress: creating tablet
I20251211 08:57:59.976047 14905 ts_tablet_manager.cc:1950] T 080d1700df4a4df09acda65c6d021d96 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:57:59.976222 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 already in progress: creating tablet
W20251211 08:57:59.977109 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f6fb2096f80a43b5a5a6230a824fceb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6fb2096f80a43b5a5a6230a824fceb7 already in progress: creating tablet
W20251211 08:57:59.979538 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet f2dc6cb4efc744e6904728ad120df934 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2dc6cb4efc744e6904728ad120df934 already in progress: creating tablet
I20251211 08:57:59.328613 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_31fcfbed4a2c4e8aa05fb42c261bf1d5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 2 got 0
I20251211 08:57:59.983347 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:57:59.986032 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 414) took 10741 ms. Trace:
I20251211 08:57:59.986382 14905 rpcz_store.cc:276] 1211 08:57:49.244253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.244444 (+ 191us) service_pool.cc:225] Handling call
1211 08:57:59.854547 (+10610103us) ts_tablet_manager.cc:1151] Deleting tablet 080d1700df4a4df09acda65c6d021d96
1211 08:57:59.854628 (+ 81us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:57:59.926048 (+ 71420us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:59.969649 (+ 43601us) tablet_metadata.cc:641] Metadata flushed
1211 08:57:59.985976 (+ 16327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10609582}
I20251211 08:57:59.987475 14714 tablet_bootstrap.cc:492] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:57:59.987872 14714 ts_tablet_manager.cc:1403] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.124s user 0.010s sys 0.026s
I20251211 08:57:59.987902 14905 tablet_replica.cc:333] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:57:59.988756 14905 raft_consensus.cc:2243] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:59.989620 14905 raft_consensus.cc:2272] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:57:59.990113 14714 raft_consensus.cc:359] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:57:59.991511 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 080d1700df4a4df09acda65c6d021d96 (table default.Table_26f4e336df6947149cb311c15bee56c1 [id=e647fb169f024aebb1f942fff0779130]) successfully deleted
I20251211 08:57:59.992304 14905 ts_tablet_manager.cc:1916] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:59.995350 14714 raft_consensus.cc:385] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:57:59.997867 14419 tablet_service.cc:1552] Processing DeleteTablet for tablet 570c7f08c46e4273bd38c32593dcf304 with delete_type TABLET_DATA_DELETED (replaced by tablet 3838fd220d034b3595f351b906188339) from {username='slave'} at 127.0.0.1:40738
W20251211 08:57:59.998349 14012 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 62) took 1083 ms (client timeout 1000 ms). Trace:
I20251211 08:57:59.998579 15941 tablet_replica.cc:333] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:57:59.998587 16044 raft_consensus.cc:493] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251211 08:57:59.998665 14012 rpcz_store.cc:269] 1211 08:57:58.915023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.915148 (+ 125us) service_pool.cc:225] Handling call
1211 08:57:58.935347 (+ 20199us) spinlock_profiling.cc:243] Waited 8.16 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:57:59.998181 (+1062834us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":9,"spinlock_wait_cycles":18775424}
I20251211 08:57:59.999255 15941 raft_consensus.cc:2243] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:57:59.998992 16044 raft_consensus.cc:515] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:57:59.999809 15941 pending_rounds.cc:70] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:57:59.999981 15941 pending_rounds.cc:77] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256490691387392 op_type: NO_OP noop_request { }
I20251211 08:58:00.000247 15941 raft_consensus.cc:2889] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:58:00.000527 15941 raft_consensus.cc:2272] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:00.000605 16044 leader_election.cc:290] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:00.002509 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "033c2c016f8f4c01adc8bdbf2885e394" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:00.002899 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet f2f095b59ecf44ffabaa0ac22212938b with delete_type TABLET_DATA_DELETED (replaced by tablet b3ad595efe5241e8a4f40b16be2ccd63) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:00.003072 14441 raft_consensus.cc:2468] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:00.003263 15941 ts_tablet_manager.cc:1916] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:00.003134 16141 tablet.cc:1722] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:58:00.004582 14542 leader_election.cc:304] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:00.004889 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet f3403116517f45c59f87fa5bda0b4071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:00.005172 15902 raft_consensus.cc:2804] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:00.005410 15902 raft_consensus.cc:493] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:00.005686 15902 raft_consensus.cc:3060] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:00.009186 14005 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55614 (request call id 604) took 1270 ms. Trace:
I20251211 08:58:00.009502 14005 rpcz_store.cc:276] 1211 08:57:58.738208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.738322 (+ 114us) service_pool.cc:225] Handling call
1211 08:57:58.758892 (+ 20570us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da87016 00007fc20b6b5b24 00007fc20b6b5ab2 00007fc20b6b59e1 00007fc20b6b595c 00007fc20b6b4c12 00007fc20998455c 00007fc209983996
1211 08:57:58.759645 (+ 753us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:58.760395 (+ 750us) catalog_manager.cc:3708] Apply alter schema
1211 08:57:58.761025 (+ 630us) catalog_manager.cc:3740] Acquired catalog manager lock
1211 08:57:58.761231 (+ 206us) catalog_manager.cc:3898] Updating metadata on disk
1211 08:57:58.814368 (+ 53137us) spinlock_profiling.cc:243] Waited 47.3 ms on lock 0x7b5400020180. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20c0276c1 00007fc20c0fbad9 00007fc20c0f16fd 00007fc20c0b2dab 00007fc20c0b2b84 00007fc20dac0671 00007fc20dac0ecf 00007fc20d974c04 00007fc20d971d2a 00007fc20d969ebb 00007fc20da87052
1211 08:57:58.875551 (+ 61183us) catalog_manager.cc:3929] Committing alterations to in-memory state
1211 08:58:00.009025 (+1133474us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:58.818560 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:58.818892 (+ 332us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:58.818912 (+ 20us) write_op.cc:435] Acquired schema lock
1211 08:57:58.818924 (+ 12us) tablet.cc:660] Decoding operations
1211 08:57:58.819133 (+ 209us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:58.819163 (+ 30us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:58.819183 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:57:58.819293 (+ 110us) tablet.cc:699] Row locks acquired
1211 08:57:58.819305 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:57:58.819387 (+ 82us) write_op.cc:270] Start()
1211 08:57:58.819450 (+ 63us) write_op.cc:276] Timestamp: P: 1765443478819525 usec, L: 0
1211 08:57:58.819464 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:57:58.819864 (+ 400us) log.cc:844] Serialized 405 byte log entry
1211 08:57:58.865407 (+ 45543us) op_driver.cc:464] REPLICATION: finished
1211 08:57:58.865609 (+ 202us) write_op.cc:301] APPLY: starting
1211 08:57:58.865661 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:58.865741 (+ 80us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:58.865817 (+ 76us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:58.865948 (+ 131us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:58.866102 (+ 154us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:57:58.866144 (+ 42us) write_op.cc:312] APPLY: finished
1211 08:57:58.866300 (+ 156us) log.cc:844] Serialized 23 byte log entry
1211 08:57:58.866443 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:58.866491 (+ 48us) write_op.cc:454] Released schema lock
1211 08:57:58.866518 (+ 27us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:58.866719 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:58.866742 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":154248704,"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":1077,"apply.run_wall_time_us":1076,"num_ops":1,"prepare.queue_time_us":4034,"prepare.run_cpu_time_us":1461,"prepare.run_wall_time_us":2318,"replication_time_us":45863}]]}
I20251211 08:57:59.995625 14714 raft_consensus.cc:740] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:00.015192 14714 consensus_queue.cc:260] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.016362 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b52de3809ace45cb9d262625fc5fb7ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:00.017612 16137 tablet.cc:1722] T 88860fa9d7e6414185f0a1cbc78b2571 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251211 08:58:00.018342 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b52de3809ace45cb9d262625fc5fb7ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:00.020015 14714 ts_tablet_manager.cc:1434] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20251211 08:58:00.022063 15902 raft_consensus.cc:515] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.022145 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_f5ae9678b61349408c4e7a5624da28ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:00.023686 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5ae9678b61349408c4e7a5624da28ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:00.024786 14714 tablet_bootstrap.cc:492] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:00.028014 15941 ts_tablet_manager.cc:1929] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:00.028614 14905 ts_tablet_manager.cc:1929] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:00.029832 15941 log.cc:1199] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/570c7f08c46e4273bd38c32593dcf304
I20251211 08:58:00.031442 15941 ts_tablet_manager.cc:1950] T 570c7f08c46e4273bd38c32593dcf304 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:00.031800 14905 log.cc:1199] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/8e05628011634149b17d298c54074d00
I20251211 08:58:00.032486 14905 ts_tablet_manager.cc:1950] T 8e05628011634149b17d298c54074d00 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:00.036151 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 570c7f08c46e4273bd38c32593dcf304 (table default.Table_f0e893169e89418ab044ae00cdb78176 [id=6ef69beb843946d080b3f6784f0e3af8]) successfully deleted
W20251211 08:58:00.038630 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f2f095b59ecf44ffabaa0ac22212938b because the tablet was not found. No further retry: Not found: Tablet not found: f2f095b59ecf44ffabaa0ac22212938b
W20251211 08:58:00.040778 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet f3403116517f45c59f87fa5bda0b4071 because the tablet was not found. No further retry: Not found: Tablet not found: f3403116517f45c59f87fa5bda0b4071
I20251211 08:58:00.044796 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "033c2c016f8f4c01adc8bdbf2885e394" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:00.045260 14441 raft_consensus.cc:3060] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:00.048327 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 415) took 10799 ms. Trace:
I20251211 08:58:00.048606 14905 rpcz_store.cc:276] 1211 08:57:49.248637 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.248830 (+ 193us) service_pool.cc:225] Handling call
1211 08:57:59.987744 (+10738914us) ts_tablet_manager.cc:1151] Deleting tablet 8e05628011634149b17d298c54074d00
1211 08:57:59.987801 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:00.007044 (+ 19243us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.028388 (+ 21344us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.048273 (+ 19885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10737945}
I20251211 08:58:00.051534 14441 raft_consensus.cc:2468] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:00.052949 14905 tablet_replica.cc:333] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:00.053263 15902 leader_election.cc:290] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:00.061496 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 8e05628011634149b17d298c54074d00 (table default.Table_885d3fb0ab5f4d0a82972d94a34c2a88 [id=18d1df6385a24d26bfb44309aa58eaa9]) successfully deleted
I20251211 08:57:59.794241 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed0e90efe651499c929b1fb992712b96" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:00.064023 14296 raft_consensus.cc:3060] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:57:59.889189 16096 raft_consensus.cc:493] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5e624bcfd154cd586ef1dedb9940063)
I20251211 08:58:00.053750 14905 raft_consensus.cc:2243] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Raft consensus shutting down.
I20251211 08:58:00.054622 14542 leader_election.cc:304] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
W20251211 08:58:00.058123 14541 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251211 08:58:00.066079 15902 raft_consensus.cc:2804] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:00.066483 15902 raft_consensus.cc:697] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:00.067013 14905 pending_rounds.cc:70] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:58:00.067276 14905 pending_rounds.cc:77] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7231256474982420480 op_type: NO_OP noop_request { }
I20251211 08:58:00.067683 14905 raft_consensus.cc:2889] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:58:00.068079 14905 raft_consensus.cc:2272] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:00.067943 15902 consensus_queue.cc:237] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.070155 14905 ts_tablet_manager.cc:1916] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:57:59.900211 14230 leader_election.cc:304] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
W20251211 08:58:00.074221 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 48575fa3f45349b9887bfc1cdfebc958 (table default.Table_a7ed3d0a6ce0464893bc9f5bb3bf6851 [id=1b3efd8519a04b13af86aa631f596f33], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 48575fa3f45349b9887bfc1cdfebc958 consensus state has no leader
I20251211 08:58:00.079115 16184 raft_consensus.cc:493] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:00.079546 16184 raft_consensus.cc:515] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.081243 16184 leader_election.cc:290] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:00.082707 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d8d84e594ea47f3bc9127cc587e8aa7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:00.084088 14382 leader_election.cc:343] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:57:59.960731 14904 tablet_replica.cc:333] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:00.089615 14904 raft_consensus.cc:2243] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:00.100008 14296 raft_consensus.cc:2468] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:58:00.086388 16142 raft_consensus.cc:493] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251211 08:58:00.116559 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e4c3a63cd7c4cc98a240e459e84a2ac consensus state has no leader
I20251211 08:58:00.109208 16195 raft_consensus.cc:493] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:00.092545 14286 raft_consensus.cc:2468] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:58:00.124502 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet ddbc5b3fad2e43b481ff733b5bad189a
I20251211 08:58:00.127341 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet ddbc5b3fad2e43b481ff733b5bad189a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:59 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:00.090788 16096 raft_consensus.cc:515] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:58:00.128902 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet ddbc5b3fad2e43b481ff733b5bad189a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddbc5b3fad2e43b481ff733b5bad189a already in progress: creating tablet
I20251211 08:58:00.117164 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b2445f66be54d3fbdc3be964b9a9132" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:00.132491 16093 raft_consensus.cc:2749] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:00.118063 16142 raft_consensus.cc:515] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.154469 16142 leader_election.cc:290] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:00.155090 14905 ts_tablet_manager.cc:1929] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251211 08:58:00.155429 14905 log.cc:1199] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/45c7d841bd3643a799a8b96e82d6d675
W20251211 08:58:00.149847 16190 log.cc:927] Time spent T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: Append to log took a long time: real 0.193s user 0.050s sys 0.000s
I20251211 08:58:00.160557 16096 leader_election.cc:290] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:00.170228 14905 ts_tablet_manager.cc:1950] T 45c7d841bd3643a799a8b96e82d6d675 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:00.171191 16183 tablet.cc:1722] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:58:00.173511 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 420) took 10682 ms. Trace:
I20251211 08:58:00.173841 14905 rpcz_store.cc:276] 1211 08:57:49.490867 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.490968 (+ 101us) service_pool.cc:225] Handling call
1211 08:58:00.052772 (+10561804us) ts_tablet_manager.cc:1151] Deleting tablet 45c7d841bd3643a799a8b96e82d6d675
1211 08:58:00.052838 (+ 66us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:00.122098 (+ 69260us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.154863 (+ 32765us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.173449 (+ 18586us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10561311}
I20251211 08:58:00.178857 15902 consensus_queue.cc:1048] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:58:00.186825 14260 consensus_meta.cc:287] T c177fcaa413a4034bc1418d66bd61e46 P 31404d5f58e548519239569b150d975b: Time spent flushing consensus metadata: real 0.642s user 0.005s sys 0.004s
I20251211 08:58:00.188701 14714 tablet_bootstrap.cc:654] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:00.206506 14904 raft_consensus.cc:2272] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:00.158149 16195 raft_consensus.cc:515] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.214309 16195 leader_election.cc:290] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:00.216948 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d8d84e594ea47f3bc9127cc587e8aa7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:00.217702 14714 tablet_bootstrap.cc:492] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:00.218098 14714 ts_tablet_manager.cc:1403] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.193s user 0.015s sys 0.000s
I20251211 08:58:00.220679 14714 raft_consensus.cc:359] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.221316 14714 raft_consensus.cc:385] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:00.221591 14714 raft_consensus.cc:740] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:00.222275 14385 leader_election.cc:304] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:00.223333 14714 consensus_queue.cc:260] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.225069 14714 ts_tablet_manager.cc:1434] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20251211 08:58:00.226239 14714 tablet_bootstrap.cc:492] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:58:00.227509 14385 leader_election.cc:343] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:00.227958 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet ddbc5b3fad2e43b481ff733b5bad189a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:59 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:00.227903 14385 leader_election.cc:304] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:00.228530 16182 raft_consensus.cc:2804] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:00.228921 16182 raft_consensus.cc:493] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:00.229252 16182 raft_consensus.cc:3060] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:00.231011 16184 raft_consensus.cc:2749] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:00.237709 16182 raft_consensus.cc:515] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.238880 16184 raft_consensus.cc:493] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:00.239277 16184 raft_consensus.cc:515] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.240949 16184 leader_election.cc:290] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:00.242839 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e36c6e2868e143be8e7f72307202a268" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:00.244292 14382 leader_election.cc:343] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:00.244686 14714 tablet_bootstrap.cc:654] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:00.245765 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 45c7d841bd3643a799a8b96e82d6d675 (table default.Table_f1453eb82ce2412c8208000d8ba7f240 [id=f08b1a30da03451ba864cb8f8e6d199b]) successfully deleted
W20251211 08:58:00.246881 14385 leader_election.cc:343] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:00.246975 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet ddbc5b3fad2e43b481ff733b5bad189a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:59 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:00.247393 14385 leader_election.cc:304] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:00.249173 16126 raft_consensus.cc:2749] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:00.252084 16182 leader_election.cc:290] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:00.252755 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c112dfcc301648cda6f0f038b1f2af83" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
W20251211 08:58:00.253921 14382 leader_election.cc:343] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:00.259346 14904 ts_tablet_manager.cc:1916] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:58:00.260895 14013 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 66) took 1666 ms (client timeout 1000 ms). Trace:
W20251211 08:58:00.261274 14013 rpcz_store.cc:269] 1211 08:57:58.594164 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.594277 (+ 113us) service_pool.cc:225] Handling call
1211 08:58:00.260397 (+1666120us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:57:58.717095 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:57:58.717463 (+ 368us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:57:58.717488 (+ 25us) write_op.cc:435] Acquired schema lock
1211 08:57:58.717501 (+ 13us) tablet.cc:660] Decoding operations
1211 08:57:58.717781 (+ 280us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:57:58.717819 (+ 38us) write_op.cc:641] Partition lock acquired for write op
1211 08:57:58.717842 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:57:58.718083 (+ 241us) tablet.cc:699] Row locks acquired
1211 08:57:58.718097 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:57:58.718206 (+ 109us) write_op.cc:270] Start()
1211 08:57:58.718275 (+ 69us) write_op.cc:276] Timestamp: P: 1765443478718287 usec, L: 0
1211 08:57:58.718288 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:57:58.718736 (+ 448us) log.cc:844] Serialized 907 byte log entry
1211 08:57:58.724888 (+ 6152us) op_driver.cc:464] REPLICATION: finished
1211 08:57:58.725256 (+ 368us) write_op.cc:301] APPLY: starting
1211 08:57:58.725301 (+ 45us) tablet.cc:1365] starting BulkCheckPresence
1211 08:57:58.725403 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
1211 08:57:58.725488 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:57:58.725719 (+ 231us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:57:58.725833 (+ 114us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:57:58.725868 (+ 35us) write_op.cc:312] APPLY: finished
1211 08:57:58.728796 (+ 2928us) log.cc:844] Serialized 29 byte log entry
1211 08:57:58.728928 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:58.728988 (+ 60us) write_op.cc:454] Released schema lock
1211 08:57:58.729019 (+ 31us) write_op.cc:341] FINISH: Updating metrics
1211 08:57:58.811798 (+ 82779us) spinlock_profiling.cc:243] Waited 80.4 ms on lock 0x7b5400020180. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20c0276c1 00007fc20c0fc28d 00007fc20c0f5312 00007fc20c0f4aa3 00007fc20c0f8cb1 00007fc20c0f8c69 00007fc20c0f8bf9 00007fc20c0f8bc1 00007fc20c0f7ded 00000000005aa2b4 00000000005aa0e9
1211 08:57:58.811966 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:57:58.812006 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":218850,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":257,"apply.run_cpu_time_us":3671,"apply.run_wall_time_us":86648,"num_ops":2,"prepare.queue_time_us":6195,"prepare.run_cpu_time_us":1923,"prepare.run_wall_time_us":1922,"raft.queue_time_us":1106,"raft.run_cpu_time_us":562,"raft.run_wall_time_us":562,"replication_time_us":6536,"spinlock_wait_cycles":184896384}]]}
I20251211 08:58:00.266116 16044 raft_consensus.cc:493] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:00.266778 16044 raft_consensus.cc:515] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.268420 15030 tablet_bootstrap.cc:492] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:00.268782 15030 ts_tablet_manager.cc:1403] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.455s user 0.015s sys 0.000s
I20251211 08:58:00.271088 16044 leader_election.cc:290] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:58:00.273059 14542 leader_election.cc:343] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: da52cace2aab4f9dacef18ec5fa56aaf
I20251211 08:58:00.277881 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet e9905227c0a844d199362a442e7d630c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:00.279345 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet f2f095b59ecf44ffabaa0ac22212938b with delete_type TABLET_DATA_DELETED (replaced by tablet b3ad595efe5241e8a4f40b16be2ccd63) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:00.280580 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet da57b72207414512b9744b283bc869fe with delete_type TABLET_DATA_DELETED (replaced by tablet 27cb4228204c4176b1a348960efd5b26) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:00.281859 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e05628011634149b17d298c54074d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:00.284655 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet e15483f539f24f1ba1b442663d66fd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:58:00.285333 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 193621us
I20251211 08:58:00.290084 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet 51c3f5eaef784e868e565f81bd51a5e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:00.291841 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet 080d1700df4a4df09acda65c6d021d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:00.293365 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet ed0e90efe651499c929b1fb992712b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:00.300607 15030 raft_consensus.cc:359] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.307796 14714 tablet_bootstrap.cc:492] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:00.308135 14714 ts_tablet_manager.cc:1403] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.082s user 0.018s sys 0.004s
I20251211 08:58:00.310863 14714 raft_consensus.cc:359] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.311585 14714 raft_consensus.cc:385] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:00.311900 14714 raft_consensus.cc:740] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:00.312520 14714 consensus_queue.cc:260] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.314112 14714 ts_tablet_manager.cc:1434] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20251211 08:58:00.315001 14714 tablet_bootstrap.cc:492] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:00.304116 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a6d23dddea0d46189b02672fbc0c3d04" } new_table_name: "default.Table_2694c137a3194291b9e34239dee11c3f" modify_external_catalogs: true
I20251211 08:58:00.305262 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_bacb3872b73a440d89e615fa9dbddfed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:00.323843 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bacb3872b73a440d89e615fa9dbddfed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:00.358299 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_20187d2b35014f1fb7329b7771d165f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:00.359447 15030 raft_consensus.cc:385] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:00.361411 15030 raft_consensus.cc:740] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:00.363070 15030 consensus_queue.cc:260] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:00.378880 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20187d2b35014f1fb7329b7771d165f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:00.385795 15030 ts_tablet_manager.cc:1434] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.117s user 0.007s sys 0.000s
I20251211 08:58:00.355420 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 419) took 10864 ms. Trace:
I20251211 08:58:00.392828 14905 rpcz_store.cc:276] 1211 08:57:49.490572 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.499006 (+ 8434us) service_pool.cc:225] Handling call
1211 08:58:00.354986 (+10855980us) ts_tablet_manager.cc:1151] Deleting tablet c03f976b926a4a6fba8b70e9df9a4160
1211 08:58:00.355050 (+ 64us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:00.355365 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10855469}
W20251211 08:58:00.395893 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a367e84c55414309bf73796eb3cce392 consensus state has no leader
I20251211 08:58:00.396883 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_a4dd5045e2db4928b29f19e423ba3188" } new_table_name: "default.Table_591e309e57604afebe45fcff55279e97" modify_external_catalogs: true
I20251211 08:58:00.398156 14009 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_cee1868d74ac4977b764abe19767825a" } modify_external_catalogs: true
I20251211 08:58:00.398708 14009 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_cee1868d74ac4977b764abe19767825a" } modify_external_catalogs: true
W20251211 08:58:00.426231 16070 lock_manager.cc:585] Waited 3 seconds to obtain row lock on key '<redacted>' (key hash 12660783061858645059) tablet 00000000000000000000000000000000 cur holder 0x7b5800045300
I20251211 08:58:00.427578 14905 tablet_replica.cc:333] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:00.428331 14905 raft_consensus.cc:2243] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:00.428815 14905 raft_consensus.cc:2272] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:58:00.442364 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10d8b7117acd41a9a6cc356ed51bb6e8 consensus state has no leader
I20251211 08:58:00.430706 14905 ts_tablet_manager.cc:1916] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:00.448112 15030 tablet_bootstrap.cc:492] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:00.450455 14263 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46844 (request call id 517) took 1022 ms. Trace:
I20251211 08:58:00.450950 14263 rpcz_store.cc:276] 1211 08:57:59.428249 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.430710 (+ 2461us) service_pool.cc:225] Handling call
1211 08:57:59.435909 (+ 5199us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1211 08:57:59.435979 (+ 70us) ts_tablet_manager.cc:682] Creating new metadata...
1211 08:57:59.583881 (+147902us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.092133 (+508252us) tablet_replica.cc:206] Creating consensus instance
1211 08:58:00.450381 (+358248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:58:00.453632 14267 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46844 (request call id 505) took 1119 ms. Trace:
I20251211 08:58:00.453893 14267 rpcz_store.cc:276] 1211 08:57:59.333869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.334011 (+ 142us) service_pool.cc:225] Handling call
1211 08:57:59.408640 (+ 74629us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1211 08:57:59.408714 (+ 74us) ts_tablet_manager.cc:682] Creating new metadata...
1211 08:57:59.638335 (+229621us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.085584 (+447249us) tablet_replica.cc:206] Creating consensus instance
1211 08:58:00.453562 (+367978us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33870}
I20251211 08:58:00.455139 14260 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46844 (request call id 516) took 1062 ms. Trace:
I20251211 08:58:00.455557 14260 rpcz_store.cc:276] 1211 08:57:59.392293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.392494 (+ 201us) service_pool.cc:225] Handling call
1211 08:57:59.421790 (+ 29296us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1211 08:57:59.437812 (+ 16022us) ts_tablet_manager.cc:682] Creating new metadata...
1211 08:57:59.544209 (+106397us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.187336 (+643127us) tablet_replica.cc:206] Creating consensus instance
1211 08:58:00.455078 (+267742us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:58:00.459084 14286 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c112dfcc301648cda6f0f038b1f2af83" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:00.459740 14286 raft_consensus.cc:3060] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:00.462993 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e36c6e2868e143be8e7f72307202a268" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:00.464377 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 55054.9us
I20251211 08:58:00.470952 14714 tablet_bootstrap.cc:654] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:00.472200 14267 tablet_service.cc:1552] Processing DeleteTablet for tablet c6d0fef50524476dbd47fc70e4bda319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:48 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:00.473266 14279 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46844 (request call id 508) took 1118 ms. Trace:
W20251211 08:58:00.475574 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 575715us
W20251211 08:58:00.476261 14230 leader_election.cc:343] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:00.476611 14230 leader_election.cc:304] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:00.473562 14279 rpcz_store.cc:276] 1211 08:57:59.355185 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.355310 (+ 125us) service_pool.cc:225] Handling call
1211 08:57:59.409953 (+ 54643us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1211 08:57:59.410017 (+ 64us) ts_tablet_manager.cc:682] Creating new metadata...
1211 08:57:59.630689 (+220672us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.226071 (+595382us) tablet_replica.cc:206] Creating consensus instance
1211 08:58:00.473220 (+247149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:58:00.479004 14266 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46844 (request call id 504) took 1148 ms. Trace:
I20251211 08:58:00.479300 14266 rpcz_store.cc:276] 1211 08:57:59.330291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.330574 (+ 283us) service_pool.cc:225] Handling call
1211 08:57:59.340451 (+ 9877us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b5800000098. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f0cff7781 00007f0f0d179ac1 00007f0f0d179972 00007f0f15f9e069 00007f0f1521d260 00007f0f15302541 00007f0f152df7d3 00007f0f1528e4c9 00007f0f13850bf4 00007f0f13850b82 00007f0f13850ab1
1211 08:57:59.405188 (+ 64737us) spinlock_profiling.cc:243] Waited 36.9 ms on lock 0x7b5800000098. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f0cff7781 00007f0f0d179dec 00007f0f0d1799ab 00007f0f15f9e069 00007f0f1521d260 00007f0f15302541 00007f0f152df7d3 00007f0f1528e4c9 00007f0f13850bf4 00007f0f13850b82 00007f0f13850ab1
1211 08:57:59.452247 (+ 47059us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1211 08:57:59.452360 (+ 113us) ts_tablet_manager.cc:682] Creating new metadata...
1211 08:57:59.570150 (+117790us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.246615 (+676465us) tablet_replica.cc:206] Creating consensus instance
1211 08:58:00.478930 (+232315us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92639872}
I20251211 08:58:00.481323 15030 tablet_bootstrap.cc:654] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:00.481913 14722 tablet_bootstrap.cc:492] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:00.481878 15902 raft_consensus.cc:493] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:00.482403 14722 ts_tablet_manager.cc:1403] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.834s user 0.013s sys 0.006s
I20251211 08:58:00.482584 15902 raft_consensus.cc:515] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.484580 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e05628011634149b17d298c54074d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:00.486119 15902 leader_election.cc:290] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:00.491376 14296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51c3f5eaef784e868e565f81bd51a5e4" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:00.501817 14714 tablet_bootstrap.cc:492] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:00.502421 14714 ts_tablet_manager.cc:1403] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.188s user 0.011s sys 0.006s
I20251211 08:58:00.505368 14714 raft_consensus.cc:359] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.506036 14714 raft_consensus.cc:385] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:00.506310 14714 raft_consensus.cc:740] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:00.506924 14714 consensus_queue.cc:260] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.509655 14714 ts_tablet_manager.cc:1434] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251211 08:58:00.516570 14296 raft_consensus.cc:2393] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c5e624bcfd154cd586ef1dedb9940063 in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:58:00.517223 14905 ts_tablet_manager.cc:1929] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:00.517576 14905 log.cc:1199] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/c341b762afd8440bb7198ca134c8dffe
I20251211 08:58:00.519589 14296 raft_consensus.cc:3060] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:00.532754 14905 ts_tablet_manager.cc:1950] T c341b762afd8440bb7198ca134c8dffe P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:00.536252 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:58:00.539883 14008 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 97) took 1793 ms (client timeout 1000 ms). Trace:
I20251211 08:58:00.540148 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 421) took 10865 ms. Trace:
I20251211 08:58:00.540486 14905 rpcz_store.cc:276] 1211 08:57:49.675118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.676434 (+ 1316us) service_pool.cc:225] Handling call
1211 08:58:00.393940 (+10717506us) ts_tablet_manager.cc:1151] Deleting tablet c341b762afd8440bb7198ca134c8dffe
1211 08:58:00.427470 (+ 33530us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:00.487276 (+ 59806us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.516983 (+ 29707us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.540096 (+ 23113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10716906}
I20251211 08:58:00.535297 14287 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed0e90efe651499c929b1fb992712b96" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:00.543633 14287 raft_consensus.cc:2393] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5e624bcfd154cd586ef1dedb9940063 in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:58:00.544450 14905 tablet_replica.cc:333] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:00.545172 14905 raft_consensus.cc:2243] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:00.545620 14905 raft_consensus.cc:2272] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:00.519209 14260 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:00.527684 16195 raft_consensus.cc:493] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5e624bcfd154cd586ef1dedb9940063)
I20251211 08:58:00.550334 16195 raft_consensus.cc:515] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.552150 16195 leader_election.cc:290] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:00.541810 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet c341b762afd8440bb7198ca134c8dffe (table default.Table_96d05b744df34836be68b4841e6c2a2b [id=86d8260b23ca4945acd9a70054b77c26]) successfully deleted
W20251211 08:58:00.552734 14008 rpcz_store.cc:269] 1211 08:57:58.746969 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.747094 (+ 125us) service_pool.cc:225] Handling call
1211 08:57:58.768489 (+ 21395us) spinlock_profiling.cc:243] Waited 7.4 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:58:00.539818 (+1771329us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":25,"spinlock_wait_cycles":17011584}
I20251211 08:58:00.556115 14905 ts_tablet_manager.cc:1916] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:00.566509 14291 raft_consensus.cc:1275] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Refusing update from remote peer c5e624bcfd154cd586ef1dedb9940063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:58:00.536628 16093 raft_consensus.cc:493] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:58:00.573287 14292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111adfd1ac1a42bdafa9416594e7e932" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:00.573784 14292 raft_consensus.cc:3060] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:00.587531 14290 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111adfd1ac1a42bdafa9416594e7e932" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:00.588229 16093 raft_consensus.cc:515] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:00.593082 16093 leader_election.cc:290] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:00.596767 14291 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c112dfcc301648cda6f0f038b1f2af83" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:00.600059 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_2543626767884e77abfbc8d233ba1db5" } new_table_name: "default.Table_cb1ee72444b64c7bb2f45e19231cf77b" modify_external_catalogs: true
I20251211 08:58:00.603163 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_e58edd45723d473d8e60bfb96dada676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:00.603374 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "033c2c016f8f4c01adc8bdbf2885e394" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:00.603892 14297 raft_consensus.cc:3060] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:58:00.604561 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e58edd45723d473d8e60bfb96dada676 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:00.615147 14287 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "033c2c016f8f4c01adc8bdbf2885e394" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:00.620837 14905 ts_tablet_manager.cc:1929] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:00.621272 14905 log.cc:1199] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/f785a3cc2e8145eebae7e9c2885b0b1e
I20251211 08:58:00.621946 14905 ts_tablet_manager.cc:1950] T f785a3cc2e8145eebae7e9c2885b0b1e P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:00.623857 14722 raft_consensus.cc:359] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.624580 14722 raft_consensus.cc:385] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:00.632063 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111adfd1ac1a42bdafa9416594e7e932" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:00.639746 14544 leader_election.cc:343] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Not found: Tablet not found: 07aef1eea7c04d9cbf35c82e45df7a16
I20251211 08:58:00.640197 14544 leader_election.cc:304] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:58:00.641270 14544 leader_election.cc:304] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:58:00.641487 15902 raft_consensus.cc:2749] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251211 08:58:00.642014 16044 raft_consensus.cc:2749] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251211 08:58:00.644136 14544 leader_election.cc:304] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:58:00.644825 16044 raft_consensus.cc:2749] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:00.645916 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_b3c68d73c7154f10856f5fe4e01e3e8a" } new_table_name: "default.Table_41ac6757b6784023894b70da19cd9641" modify_external_catalogs: true
I20251211 08:58:00.646001 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "578404732eef4e51b66b6aebffa87cdb" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:00.646684 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 422) took 10679 ms. Trace:
I20251211 08:58:00.647732 14905 rpcz_store.cc:276] 1211 08:57:49.966825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:49.966918 (+ 93us) service_pool.cc:225] Handling call
1211 08:58:00.544276 (+10577358us) ts_tablet_manager.cc:1151] Deleting tablet f785a3cc2e8145eebae7e9c2885b0b1e
1211 08:58:00.544333 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:00.595012 (+ 50679us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.620591 (+ 25579us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.646634 (+ 26043us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10473574}
I20251211 08:58:00.649569 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet f785a3cc2e8145eebae7e9c2885b0b1e (table default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]) successfully deleted
I20251211 08:58:00.650923 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 424) took 10467 ms. Trace:
W20251211 08:58:00.651733 14008 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 605ccab400ed4557bfd7c61cba4448e2 consensus state has no leader
I20251211 08:58:00.655668 14722 raft_consensus.cc:740] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:00.656257 14722 consensus_queue.cc:260] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.656150 14905 rpcz_store.cc:276] 1211 08:57:50.183800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:50.183903 (+ 103us) service_pool.cc:225] Handling call
1211 08:58:00.650569 (+10466666us) ts_tablet_manager.cc:1151] Deleting tablet e62ae5b540aa4680b4d18fee29c39e64
1211 08:58:00.650629 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:00.650878 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10425138}
I20251211 08:58:00.659242 14905 tablet_replica.cc:333] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:58:00.660087 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet e62ae5b540aa4680b4d18fee29c39e64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e62ae5b540aa4680b4d18fee29c39e64 already in progress: creating tablet
I20251211 08:58:00.627722 14292 raft_consensus.cc:2468] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:00.661250 14905 raft_consensus.cc:2243] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251211 08:58:00.628212 16201 log.cc:927] Time spent T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b: Append to log took a long time: real 0.325s user 0.001s sys 0.000s
I20251211 08:58:00.664963 16183 tablet.cc:1722] T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251211 08:58:00.667758 14007 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7831cd0a563a4f239b7f2d1c8c333b61 consensus state has no leader
I20251211 08:58:00.670058 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_4f8bc056ef464b31b042453e664e7586"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:00.672632 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f8bc056ef464b31b042453e664e7586 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:00.674649 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_1e0510e1cd434ce0838cdf8b1915fe66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:00.678645 14007 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e0510e1cd434ce0838cdf8b1915fe66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:00.683141 16044 consensus_queue.cc:1048] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:58:00.686635 14905 raft_consensus.cc:2272] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:00.688776 14905 ts_tablet_manager.cc:1916] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:58:00.708500 14544 leader_election.cc:343] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Not found: Tablet not found: ccce5a446ac9460c9c9b7de72d875d95
I20251211 08:58:00.708926 14544 leader_election.cc:304] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
W20251211 08:58:00.709682 14544 raft_consensus.cc:2686] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251211 08:58:00.714529 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c112dfcc301648cda6f0f038b1f2af83" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:00.716116 14714 tablet_bootstrap.cc:492] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:00.716711 14441 raft_consensus.cc:2393] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5e624bcfd154cd586ef1dedb9940063 in current term 1: Already voted for candidate b276f77f87254a5d909c5ab940a39256 in this term.
I20251211 08:58:00.722970 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_36432f79cf0a4a53bdc732d932422846"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:00.724354 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36432f79cf0a4a53bdc732d932422846 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:00.730158 14260 tablet_service.cc:1552] Processing DeleteTablet for tablet 00e5c4bf71b44abca66b0e94995c533e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:00.731285 14260 tablet_service.cc:1552] Processing DeleteTablet for tablet f3403116517f45c59f87fa5bda0b4071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:00.735375 14722 ts_tablet_manager.cc:1434] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.253s user 0.004s sys 0.002s
I20251211 08:58:00.775434 14286 raft_consensus.cc:2468] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:58:00.775920 16203 raft_consensus.cc:2749] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:00.788779 16093 consensus_queue.cc:1048] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:00.797312 14714 tablet_bootstrap.cc:654] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:00.800348 14297 raft_consensus.cc:2468] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
W20251211 08:58:00.800630 14543 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251211 08:58:00.811517 14722 tablet_bootstrap.cc:492] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:58:00.814848 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 102304us
W20251211 08:58:00.815388 14544 leader_election.cc:343] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Not found: Tablet not found: da52cace2aab4f9dacef18ec5fa56aaf
I20251211 08:58:00.815706 14544 leader_election.cc:304] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:58:00.816303 16044 raft_consensus.cc:2749] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251211 08:58:00.828454 15030 tablet_bootstrap.cc:492] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:00.828861 15030 ts_tablet_manager.cc:1403] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.381s user 0.017s sys 0.069s
W20251211 08:58:00.833901 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 98470.3us
I20251211 08:58:00.834679 16093 raft_consensus.cc:493] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:00.835220 16093 raft_consensus.cc:515] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.836133 14544 leader_election.cc:304] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:58:00.836766 15902 raft_consensus.cc:3060] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:00.837332 16093 leader_election.cc:290] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:00.842340 16184 raft_consensus.cc:493] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:00.842862 16184 raft_consensus.cc:515] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.843261 15030 raft_consensus.cc:359] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.843914 15030 raft_consensus.cc:385] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:00.844144 15030 raft_consensus.cc:740] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:00.844551 15902 raft_consensus.cc:2749] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:00.844684 15030 consensus_queue.cc:260] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.846522 15030 ts_tablet_manager.cc:1434] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.017s user 0.004s sys 0.002s
I20251211 08:58:00.850205 16184 leader_election.cc:290] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:00.859393 15030 tablet_bootstrap.cc:492] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:00.872344 15030 tablet_bootstrap.cc:654] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:00.879339 15902 consensus_queue.cc:1048] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:00.881001 15902 consensus_queue.cc:1048] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:00.900972 15030 tablet_bootstrap.cc:492] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:00.901481 15030 ts_tablet_manager.cc:1403] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.005s
I20251211 08:58:00.903190 14013 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_321194b0f71d457e81e14c810650b486" } new_table_name: "default.Table_ae8d82f4355240188c054f001b2c05a4" modify_external_catalogs: true
I20251211 08:58:00.903723 15030 raft_consensus.cc:359] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.904487 15030 raft_consensus.cc:385] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:00.904759 15030 raft_consensus.cc:740] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:00.905369 15030 consensus_queue.cc:260] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:00.905711 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_fb66cfb783c94088a6a035c6a95b5f43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:00.907367 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb66cfb783c94088a6a035c6a95b5f43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:00.907513 15030 ts_tablet_manager.cc:1434] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20251211 08:58:00.908444 15030 tablet_bootstrap.cc:492] T 0fdc020f21aa4f35880a435d7cfb605d P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:00.910434 14007 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_31fcfbed4a2c4e8aa05fb42c261bf1d5" } new_table_name: "default.Table_cd6e057b26b443ae9e79158e9350dce5" modify_external_catalogs: true
I20251211 08:58:00.914487 14722 tablet_bootstrap.cc:654] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:00.917003 15030 tablet_bootstrap.cc:654] T 0fdc020f21aa4f35880a435d7cfb605d P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:00.932327 14655 heartbeater.cc:344] Connected to a master server at 127.13.146.62:39015
W20251211 08:58:00.934989 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 49425.6us
I20251211 08:58:00.938202 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fdc020f21aa4f35880a435d7cfb605d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:00.939586 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41d6ef975a47489385791065bb8f9f46" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:00.970839 14904 ts_tablet_manager.cc:1929] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:00.974713 15902 consensus_queue.cc:1048] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:58:00.979164 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c0bdd2d0d9f4221bf13812bb7a97025" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:00.979666 14607 raft_consensus.cc:2468] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:58:00.980099 14904 log.cc:1199] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/f785a3cc2e8145eebae7e9c2885b0b1e
I20251211 08:58:00.983754 14904 ts_tablet_manager.cc:1950] T f785a3cc2e8145eebae7e9c2885b0b1e P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:00.984431 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 460) took 5190 ms. Trace:
I20251211 08:58:00.984686 14904 rpcz_store.cc:276] 1211 08:57:55.794159 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.822868 (+ 28709us) service_pool.cc:225] Handling call
1211 08:57:59.895349 (+4072481us) ts_tablet_manager.cc:1151] Deleting tablet f785a3cc2e8145eebae7e9c2885b0b1e
1211 08:57:59.960615 (+ 65266us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:00.206213 (+245598us) spinlock_profiling.cc:243] Waited 39.1 ms on lock 0x7b58001004d0. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f141cc5a1 00007f0f141bf0da 00007f0f14241177 00007f0f1426b005 00007f0f1468d2ec 00007f0f152e431a 00007f0f15314f80 00007f0f152fb54d 00007f0f152fb4f9 00007f0f152fb489 00007f0f152fb451
1211 08:58:00.772343 (+566130us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.970599 (+198256us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:00.984381 (+ 13782us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89920384,"tablet-delete.queue_time_us":4043344}
I20251211 08:58:00.986003 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet f785a3cc2e8145eebae7e9c2885b0b1e (table default.Table_7ff0193278184b638a2f8e40fd323397 [id=38c4725adabc4d9885147edca1cf947b]) successfully deleted
I20251211 08:58:00.990473 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111adfd1ac1a42bdafa9416594e7e932" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
W20251211 08:58:00.992893 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 306416us
I20251211 08:58:00.995858 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "578404732eef4e51b66b6aebffa87cdb" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:01.000240 14714 tablet_bootstrap.cc:492] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:01.000658 14714 ts_tablet_manager.cc:1403] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.285s user 0.009s sys 0.004s
I20251211 08:58:01.010984 14905 ts_tablet_manager.cc:1929] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:01.011405 14905 log.cc:1199] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/ccce5a446ac9460c9c9b7de72d875d95
I20251211 08:58:01.012110 14905 ts_tablet_manager.cc:1950] T ccce5a446ac9460c9c9b7de72d875d95 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:58:01.015261 14230 consensus_peers.cc:597] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 080d1700df4a4df09acda65c6d021d96. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:01.016166 14230 leader_election.cc:343] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:58:01.029006 14230 leader_election.cc:343] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: c341b762afd8440bb7198ca134c8dffe
I20251211 08:58:01.032578 14722 tablet_bootstrap.cc:492] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
W20251211 08:58:01.033322 16218 log.cc:927] Time spent T f5c5b1aebde94dea81cb37a6a6998070 P 31404d5f58e548519239569b150d975b: Append to log took a long time: real 0.073s user 0.002s sys 0.007s
I20251211 08:58:01.039119 14722 ts_tablet_manager.cc:1403] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.228s user 0.012s sys 0.001s
W20251211 08:58:01.039856 14385 leader_election.cc:343] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.041155 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fdc020f21aa4f35880a435d7cfb605d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:01.041972 14441 raft_consensus.cc:2468] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:58:01.043455 16215 tablet.cc:1722] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251211 08:58:01.043803 14904 tablet_replica.cc:333] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:01.044335 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41d6ef975a47489385791065bb8f9f46" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:01.044994 14904 raft_consensus.cc:2243] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:01.045868 14904 raft_consensus.cc:2272] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:01.048945 14722 raft_consensus.cc:359] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.049703 14722 raft_consensus.cc:385] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:01.049944 14722 raft_consensus.cc:740] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:01.061358 14722 consensus_queue.cc:260] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.061057 16183 tablet.cc:1722] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:58:01.066787 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_9d37990ab8b1472a9b0390875ea8e433"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:01.068492 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d37990ab8b1472a9b0390875ea8e433 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:01.078089 14722 ts_tablet_manager.cc:1434] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20251211 08:58:01.079025 14722 tablet_bootstrap.cc:492] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:01.095064 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c02cb6ec01041dba118683a57aecde2" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:01.040341 14385 leader_election.cc:304] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
W20251211 08:58:01.107712 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 67931.6us
I20251211 08:58:01.111007 16223 raft_consensus.cc:2749] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:01.118093 16224 raft_consensus.cc:493] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:01.118623 16224 raft_consensus.cc:515] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:01.121104 16224 leader_election.cc:290] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:01.133813 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c0bdd2d0d9f4221bf13812bb7a97025" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:01.134599 14441 raft_consensus.cc:2468] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:58:01.139776 14714 raft_consensus.cc:359] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.140429 14714 raft_consensus.cc:385] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:01.140681 14714 raft_consensus.cc:740] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:01.141268 14714 consensus_queue.cc:260] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:01.132747 14231 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251211 08:58:01.155556 16239 tablet.cc:1722] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:58:01.108551 14385 leader_election.cc:304] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.157931 14904 ts_tablet_manager.cc:1916] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:58:01.159737 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet e90daf0f780a467e9c900b91c1615cb2 (table default.Table_ea127c2426eb46ee891ec39320734259 [id=afa43e27480743738f123b2b0660f51e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e90daf0f780a467e9c900b91c1615cb2 consensus state has no leader
I20251211 08:58:01.159865 14231 leader_election.cc:304] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.161295 14231 leader_election.cc:304] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.163699 14231 leader_election.cc:304] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
W20251211 08:58:01.177109 14231 consensus_peers.cc:597] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3403116517f45c59f87fa5bda0b4071. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:01.177850 14231 leader_election.cc:304] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.179625 14714 ts_tablet_manager.cc:1434] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.179s user 0.006s sys 0.001s
I20251211 08:58:01.202936 16182 raft_consensus.cc:493] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:01.203477 16182 raft_consensus.cc:515] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:01.205404 16182 leader_election.cc:290] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:01.182314 16096 raft_consensus.cc:2804] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:01.181838 16203 raft_consensus.cc:2749] T 578404732eef4e51b66b6aebffa87cdb P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:58:01.207693 16096 raft_consensus.cc:493] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:01.208055 16096 raft_consensus.cc:3060] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:01.226907 16203 raft_consensus.cc:2749] T 41d6ef975a47489385791065bb8f9f46 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:58:01.226347 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7ed565fc5cb943fb85b6cf27e3c57e6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:01.230072 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_617b1cf1914744c1bc22abe21985028d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:01.233116 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_617b1cf1914744c1bc22abe21985028d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:58:01.240690 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ed565fc5cb943fb85b6cf27e3c57e6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:01.259799 16096 raft_consensus.cc:515] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.262045 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_14d314e70b024f78ad6b5d01e93fc113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:01.263612 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14d314e70b024f78ad6b5d01e93fc113 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:58:01.303506 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 195152us
I20251211 08:58:01.305380 16224 raft_consensus.cc:2804] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:01.306166 16224 raft_consensus.cc:697] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:58:01.307188 16224 consensus_queue.cc:237] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.310530 16096 leader_election.cc:290] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:01.318835 14714 tablet_bootstrap.cc:492] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:01.320183 16093 raft_consensus.cc:2749] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:01.324055 14714 tablet_bootstrap.cc:654] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:01.332209 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac6be204692a44fab2db551e79237299" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:01.337253 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 159562us
I20251211 08:58:01.337922 14714 tablet_bootstrap.cc:492] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
W20251211 08:58:01.337903 14231 leader_election.cc:343] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: c341b762afd8440bb7198ca134c8dffe
I20251211 08:58:01.338389 14714 ts_tablet_manager.cc:1403] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20251211 08:58:01.338267 14231 leader_election.cc:304] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
W20251211 08:58:01.346717 14382 leader_election.cc:343] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:58:01.348802 14382 leader_election.cc:343] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.356967 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cc3373962574253868ababbbe78e57d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:01.359944 14230 leader_election.cc:304] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:01.361493 16203 raft_consensus.cc:2749] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:58:01.362550 16203 raft_consensus.cc:2804] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:01.362926 16203 raft_consensus.cc:493] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:01.363222 16203 raft_consensus.cc:3060] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:01.368296 16219 consensus_queue.cc:1048] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:01.372701 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 431) took 10620 ms. Trace:
I20251211 08:58:01.373003 16203 raft_consensus.cc:515] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.373133 14905 rpcz_store.cc:276] 1211 08:57:50.751847 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:50.751968 (+ 121us) service_pool.cc:225] Handling call
1211 08:58:00.658644 (+9906676us) ts_tablet_manager.cc:1151] Deleting tablet ccce5a446ac9460c9c9b7de72d875d95
1211 08:58:00.658702 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:00.848967 (+190265us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:01.010722 (+161755us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:01.372641 (+361919us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24783,"tablet-delete.queue_time_us":9906155}
I20251211 08:58:01.379035 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fdc020f21aa4f35880a435d7cfb605d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:01.379541 14440 raft_consensus.cc:3060] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:01.380895 16203 leader_election.cc:290] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:01.380573 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_74ae14e8cc744925a5f481dcf5eac8ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:01.382441 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74ae14e8cc744925a5f481dcf5eac8ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:01.388257 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c02cb6ec01041dba118683a57aecde2" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:01.389667 14440 raft_consensus.cc:2468] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:58:01.393797 15030 tablet_bootstrap.cc:492] T 0fdc020f21aa4f35880a435d7cfb605d P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:01.394219 15030 ts_tablet_manager.cc:1403] T 0fdc020f21aa4f35880a435d7cfb605d P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.486s user 0.130s sys 0.224s
I20251211 08:58:01.395856 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 450) took 9936 ms. Trace:
I20251211 08:58:01.396205 14905 rpcz_store.cc:276] 1211 08:57:51.459929 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.460365 (+ 436us) service_pool.cc:225] Handling call
1211 08:58:01.374373 (+9914008us) ts_tablet_manager.cc:1151] Deleting tablet 63492ba35e26450f9645379a6501d544
1211 08:58:01.374438 (+ 65us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:01.395796 (+ 21358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9913404}
I20251211 08:58:01.396605 15030 raft_consensus.cc:359] T 0fdc020f21aa4f35880a435d7cfb605d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.397230 15030 raft_consensus.cc:385] T 0fdc020f21aa4f35880a435d7cfb605d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:01.397460 15030 raft_consensus.cc:740] T 0fdc020f21aa4f35880a435d7cfb605d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:01.397944 15030 consensus_queue.cc:260] T 0fdc020f21aa4f35880a435d7cfb605d P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.399796 15030 ts_tablet_manager.cc:1434] T 0fdc020f21aa4f35880a435d7cfb605d P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20251211 08:58:01.411063 16183 tablet.cc:1722] T 88860fa9d7e6414185f0a1cbc78b2571 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251211 08:58:01.417186 14382 leader_election.cc:343] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.417876 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac6be204692a44fab2db551e79237299" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:01.419925 16093 raft_consensus.cc:493] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5e624bcfd154cd586ef1dedb9940063)
I20251211 08:58:01.420449 16093 raft_consensus.cc:515] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:01.422256 16093 leader_election.cc:290] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:01.425010 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c0bdd2d0d9f4221bf13812bb7a97025" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:01.425544 14449 raft_consensus.cc:3060] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:01.434018 14449 raft_consensus.cc:2468] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
W20251211 08:58:01.437572 14230 leader_election.cc:343] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: e9905227c0a844d199362a442e7d630c
I20251211 08:58:01.442308 14722 tablet_bootstrap.cc:654] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:01.446971 14231 leader_election.cc:304] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:58:01.447722 16093 raft_consensus.cc:2804] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:01.448195 16093 raft_consensus.cc:697] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:58:01.449052 16093 consensus_queue.cc:237] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.458235 14008 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_df875ee987f64781b50ea89729e90a02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:01.459236 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet ccce5a446ac9460c9c9b7de72d875d95 (table default.Table_afa03065224d44a3a321c94b59f12587 [id=6d0af44f2d4542168eb2af3ead7fe1c4]) successfully deleted
W20251211 08:58:01.459784 14008 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df875ee987f64781b50ea89729e90a02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:01.467615 14714 raft_consensus.cc:359] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.468276 14714 raft_consensus.cc:385] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:01.468529 14714 raft_consensus.cc:740] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:01.469193 14714 consensus_queue.cc:260] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.470554 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cc3373962574253868ababbbe78e57d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:01.491528 16223 raft_consensus.cc:493] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:01.492014 16223 raft_consensus.cc:515] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:01.494006 16223 leader_election.cc:290] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:58:01.495049 14385 leader_election.cc:343] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.495422 14385 leader_election.cc:304] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.497686 14714 ts_tablet_manager.cc:1434] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.159s user 0.002s sys 0.005s
I20251211 08:58:01.502485 14006 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_3ae97e7602364fe19912c59bff54d893"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:01.504108 14006 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ae97e7602364fe19912c59bff54d893 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:01.504842 16219 raft_consensus.cc:2749] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:58:01.505077 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 58185.3us
I20251211 08:58:01.508052 16223 consensus_queue.cc:1048] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:01.509256 14904 ts_tablet_manager.cc:1929] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:01.509625 14904 log.cc:1199] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/080d1700df4a4df09acda65c6d021d96
I20251211 08:58:01.510344 14904 ts_tablet_manager.cc:1950] T 080d1700df4a4df09acda65c6d021d96 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:01.510651 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c0bdd2d0d9f4221bf13812bb7a97025" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:58:01.511193 14605 raft_consensus.cc:3060] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:01.511677 15030 tablet_bootstrap.cc:492] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:01.505661 14231 leader_election.cc:304] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
W20251211 08:58:01.518579 14231 leader_election.cc:343] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: e9905227c0a844d199362a442e7d630c
I20251211 08:58:01.518967 14231 leader_election.cc:304] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.519730 16241 raft_consensus.cc:2804] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:01.520153 16241 raft_consensus.cc:697] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:58:01.510681 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0fdc020f21aa4f35880a435d7cfb605d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:58:01.520855 16241 consensus_queue.cc:237] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.521682 14599 raft_consensus.cc:3060] T 0fdc020f21aa4f35880a435d7cfb605d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:01.522941 16096 raft_consensus.cc:2749] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:58:01.522981 14605 raft_consensus.cc:2468] T 7c0bdd2d0d9f4221bf13812bb7a97025 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:58:01.527141 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 453) took 5734 ms. Trace:
I20251211 08:58:01.527443 14904 rpcz_store.cc:276] 1211 08:57:55.792379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.847239 (+ 54860us) service_pool.cc:225] Handling call
1211 08:58:01.026840 (+5179601us) ts_tablet_manager.cc:1151] Deleting tablet 080d1700df4a4df09acda65c6d021d96
1211 08:58:01.035535 (+ 8695us) spinlock_profiling.cc:243] Waited 7 ms on lock 0x7b5800000098. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f0cff7781 00007f0f0d179ac1 00007f0f0d17a0f2 00007f0f15fc7419 00007f0f1521d108 00007f0f152e39a9 00007f0f152e3fa3 00007f0f15314f80 00007f0f152fb54d 00007f0f152fb4f9 00007f0f152fb489
1211 08:58:01.043632 (+ 8097us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:01.198448 (+154816us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:01.509038 (+310590us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:01.527089 (+ 18051us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16095104,"tablet-delete.queue_time_us":5136164}
I20251211 08:58:01.529613 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 080d1700df4a4df09acda65c6d021d96 (table default.Table_26f4e336df6947149cb311c15bee56c1 [id=e647fb169f024aebb1f942fff0779130]) successfully deleted
W20251211 08:58:01.529824 14044 debug-util.cc:398] Leaking SignalData structure 0x7b08000f0180 after lost signal to thread 13979
W20251211 08:58:01.530516 14044 debug-util.cc:398] Leaking SignalData structure 0x7b08000fd320 after lost signal to thread 14047
I20251211 08:58:01.537173 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 451) took 10077 ms. Trace:
I20251211 08:58:01.529779 15030 tablet_bootstrap.cc:654] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:01.538551 14905 rpcz_store.cc:276] 1211 08:57:51.460248 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.461014 (+ 766us) service_pool.cc:225] Handling call
1211 08:58:01.396993 (+9935979us) ts_tablet_manager.cc:1151] Deleting tablet 29869ac852e64489973feb8b0d86ab57
1211 08:58:01.451940 (+ 54947us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:01.537118 (+ 85178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9935506}
I20251211 08:58:01.542326 14722 tablet_bootstrap.cc:492] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:01.542794 14722 ts_tablet_manager.cc:1403] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.464s user 0.010s sys 0.000s
I20251211 08:58:01.545324 14722 raft_consensus.cc:359] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:01.545799 14385 leader_election.cc:343] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.546244 14385 leader_election.cc:304] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.549472 14599 raft_consensus.cc:2468] T 0fdc020f21aa4f35880a435d7cfb605d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:58:01.550350 16224 raft_consensus.cc:2749] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:01.550527 14905 tablet_replica.cc:333] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:01.551379 14905 raft_consensus.cc:2243] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:01.551982 14905 raft_consensus.cc:2272] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:01.553615 14905 ts_tablet_manager.cc:1916] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:01.553779 14714 tablet_bootstrap.cc:492] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:01.573704 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6874fb018bc844b3aeeab7db8fc53e9a" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:01.576623 16246 consensus_queue.cc:1048] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:58:01.577026 14385 leader_election.cc:343] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.577409 14385 leader_election.cc:304] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.583539 16224 raft_consensus.cc:2749] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:58:01.596560 14382 leader_election.cc:343] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.599643 14714 tablet_bootstrap.cc:654] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:01.604811 15030 tablet_bootstrap.cc:492] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:01.605271 15030 ts_tablet_manager.cc:1403] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.094s user 0.011s sys 0.003s
W20251211 08:58:01.606083 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 63492ba35e26450f9645379a6501d544 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63492ba35e26450f9645379a6501d544 already in progress: creating tablet
W20251211 08:58:01.609117 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 29869ac852e64489973feb8b0d86ab57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29869ac852e64489973feb8b0d86ab57 already in progress: creating tablet
I20251211 08:58:01.622100 14714 tablet_bootstrap.cc:492] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:01.622555 14714 ts_tablet_manager.cc:1403] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.008s
I20251211 08:58:01.625211 14714 raft_consensus.cc:359] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.625870 14714 raft_consensus.cc:385] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:01.626121 14714 raft_consensus.cc:740] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:01.607419 16246 raft_consensus.cc:493] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:01.627271 16246 raft_consensus.cc:515] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.629976 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b2445f66be54d3fbdc3be964b9a9132" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:01.630498 14440 raft_consensus.cc:2468] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:01.631546 14542 leader_election.cc:304] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:01.632323 15902 raft_consensus.cc:2804] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:01.632366 16246 leader_election.cc:290] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:01.632663 15902 raft_consensus.cc:493] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:01.632948 15902 raft_consensus.cc:3060] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:01.569262 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 463) took 5712 ms. Trace:
I20251211 08:58:01.597779 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6874fb018bc844b3aeeab7db8fc53e9a" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:01.635002 15030 raft_consensus.cc:359] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:01.635650 15030 raft_consensus.cc:385] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:01.634892 14722 raft_consensus.cc:385] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:01.634905 14904 rpcz_store.cc:276] 1211 08:57:55.856821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.856950 (+ 129us) service_pool.cc:225] Handling call
1211 08:58:01.568947 (+5711997us) ts_tablet_manager.cc:1151] Deleting tablet 785a5edbda984467b897ee1bbaf1430c
1211 08:58:01.568989 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:01.569206 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5668103}
I20251211 08:58:01.635954 15030 raft_consensus.cc:740] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:01.636404 14722 raft_consensus.cc:740] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:01.637100 14722 consensus_queue.cc:260] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:01.638224 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 785a5edbda984467b897ee1bbaf1430c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 785a5edbda984467b897ee1bbaf1430c already in progress: creating tablet
I20251211 08:58:01.638890 14722 ts_tablet_manager.cc:1434] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.096s user 0.006s sys 0.000s
W20251211 08:58:01.638975 14385 leader_election.cc:343] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.639330 14385 leader_election.cc:304] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.640432 16219 raft_consensus.cc:2749] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:01.645769 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b2445f66be54d3fbdc3be964b9a9132" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:01.647722 14904 tablet_replica.cc:333] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:01.660755 14904 raft_consensus.cc:2243] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:01.660573 15902 raft_consensus.cc:515] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.661243 14904 raft_consensus.cc:2272] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:01.663197 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b2445f66be54d3fbdc3be964b9a9132" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:01.663403 14904 ts_tablet_manager.cc:1916] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:01.663690 14440 raft_consensus.cc:3060] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:01.671381 14256 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:46838 (request call id 73) took 1580 ms. Trace:
I20251211 08:58:01.671739 14256 rpcz_store.cc:276] 1211 08:58:00.091425 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.091557 (+ 132us) service_pool.cc:225] Handling call
1211 08:58:00.124990 (+ 33433us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b5800000098. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f0cff7781 00007f0f0d179ac1 00007f0f0d179e3e 00007f0f16551049 00007f0f1521d560 00007f0f152e5d61 00007f0f152e5f20 00007f0f1529d132 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1
1211 08:58:01.180685 (+1055695us) spinlock_profiling.cc:243] Waited 77.7 ms on lock 0x7b5400181ad4. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f141cc5a1 00007f0f1425cb78 00007f0f1529d20d 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:58:01.299889 (+119204us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b5c0007150c. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f146046c1 00007f0f147a4e19 00007f0f1469033c 00007f0f1529d1e0 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996
1211 08:58:01.671322 (+371433us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":261139584}
I20251211 08:58:01.676437 15902 leader_election.cc:290] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:01.680840 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b2445f66be54d3fbdc3be964b9a9132" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:58:01.681507 14544 leader_election.cc:343] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.683288 14440 raft_consensus.cc:2468] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:01.684237 14542 leader_election.cc:304] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:01.686251 14714 consensus_queue.cc:260] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.689566 16243 raft_consensus.cc:2804] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:01.690064 16243 raft_consensus.cc:697] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:01.690838 16243 consensus_queue.cc:237] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.691484 14722 tablet_bootstrap.cc:492] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:58:01.701628 14544 leader_election.cc:343] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.710600 14714 ts_tablet_manager.cc:1434] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.088s user 0.005s sys 0.000s
I20251211 08:58:01.719895 15030 consensus_queue.cc:260] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:01.722534 15030 ts_tablet_manager.cc:1434] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.090s user 0.004s sys 0.000s
I20251211 08:58:01.723433 15030 tablet_bootstrap.cc:492] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:01.735330 14714 tablet_bootstrap.cc:492] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:01.746328 14257 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:46838 (request call id 72) took 1675 ms. Trace:
I20251211 08:58:01.746696 14257 rpcz_store.cc:276] 1211 08:58:00.070752 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.070871 (+ 119us) service_pool.cc:225] Handling call
1211 08:58:00.114487 (+ 43616us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b54000c4c54. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f141cc5a1 00007f0f1425cb78 00007f0f1529d20d 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:58:01.284343 (+1169856us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b54001a1b54. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f141cc5a1 00007f0f1425cb78 00007f0f1529d20d 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:58:01.746261 (+461918us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":468318592}
I20251211 08:58:01.760320 14714 tablet_bootstrap.cc:654] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:01.777405 14904 ts_tablet_manager.cc:1929] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251211 08:58:01.777798 14904 log.cc:1199] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/da57b72207414512b9744b283bc869fe
I20251211 08:58:01.778469 14904 ts_tablet_manager.cc:1950] T da57b72207414512b9744b283bc869fe P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:01.790141 14714 tablet_bootstrap.cc:492] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:01.790525 14714 ts_tablet_manager.cc:1403] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.007s
I20251211 08:58:01.792631 14714 raft_consensus.cc:359] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.794088 14714 raft_consensus.cc:385] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:01.794319 14714 raft_consensus.cc:740] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:01.794994 14714 consensus_queue.cc:260] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.799423 16246 raft_consensus.cc:493] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:01.799882 16246 raft_consensus.cc:515] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:01.801537 16246 leader_election.cc:290] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:01.803495 14722 tablet_bootstrap.cc:654] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:01.804617 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6874fb018bc844b3aeeab7db8fc53e9a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:01.805212 14440 raft_consensus.cc:2468] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:01.809026 14714 ts_tablet_manager.cc:1434] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.018s user 0.002s sys 0.003s
I20251211 08:58:01.806705 14542 leader_election.cc:304] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:01.809914 14714 tablet_bootstrap.cc:492] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:01.814790 14714 tablet_bootstrap.cc:654] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:01.814837 16246 raft_consensus.cc:2804] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:01.815182 16246 raft_consensus.cc:493] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:01.815495 16246 raft_consensus.cc:3060] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:01.819702 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6874fb018bc844b3aeeab7db8fc53e9a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:01.821041 14544 leader_election.cc:343] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.827699 15030 tablet_bootstrap.cc:654] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:01.830895 14905 ts_tablet_manager.cc:1929] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:01.831280 14905 log.cc:1199] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/07aef1eea7c04d9cbf35c82e45df7a16
I20251211 08:58:01.832268 14905 ts_tablet_manager.cc:1950] T 07aef1eea7c04d9cbf35c82e45df7a16 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:01.847585 16247 raft_consensus.cc:493] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:01.851615 16247 raft_consensus.cc:515] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.853273 16247 leader_election.cc:290] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:01.855262 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 452) took 10370 ms. Trace:
I20251211 08:58:01.855577 14905 rpcz_store.cc:276] 1211 08:57:51.484722 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.484858 (+ 136us) service_pool.cc:225] Handling call
1211 08:58:01.541401 (+10056543us) ts_tablet_manager.cc:1151] Deleting tablet 07aef1eea7c04d9cbf35c82e45df7a16
1211 08:58:01.550381 (+ 8980us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:01.691203 (+140822us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:01.830638 (+139435us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:01.855214 (+ 24576us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10056035}
W20251211 08:58:01.856230 14231 leader_election.cc:343] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: f2f095b59ecf44ffabaa0ac22212938b
I20251211 08:58:01.856658 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 07aef1eea7c04d9cbf35c82e45df7a16 (table default.Table_d40cb8a6b4974e2e9ff982e893a2c552 [id=d6acd0e03f2743349b345927f5ce7a6d]) successfully deleted
I20251211 08:58:01.857771 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 456) took 6064 ms. Trace:
W20251211 08:58:01.857903 14230 leader_election.cc:343] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: f2f095b59ecf44ffabaa0ac22212938b
I20251211 08:58:01.858250 14230 leader_election.cc:304] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.859110 16241 raft_consensus.cc:2749] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:01.858063 14904 rpcz_store.cc:276] 1211 08:57:55.793147 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.822804 (+ 29657us) service_pool.cc:225] Handling call
1211 08:58:01.636325 (+5813521us) ts_tablet_manager.cc:1151] Deleting tablet da57b72207414512b9744b283bc869fe
1211 08:58:01.647601 (+ 11276us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:01.670253 (+ 22652us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:01.776979 (+106726us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:01.857720 (+ 80741us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12265,"tablet-delete.queue_time_us":5727930}
I20251211 08:58:01.861177 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet da57b72207414512b9744b283bc869fe (table default.Table_2f325f51dd9f433f98cfef7169cdbe0c [id=2ff95cefc4a545398f979e2bd0321f5a]) successfully deleted
I20251211 08:58:01.862779 16246 raft_consensus.cc:515] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:01.865097 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6874fb018bc844b3aeeab7db8fc53e9a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:01.866302 14904 tablet_replica.cc:333] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b: stopping tablet replica
W20251211 08:58:01.866240 14544 leader_election.cc:343] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.867151 14904 raft_consensus.cc:2243] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:01.867542 14904 raft_consensus.cc:2272] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:01.869594 14904 ts_tablet_manager.cc:1916] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:01.874302 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6874fb018bc844b3aeeab7db8fc53e9a" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:01.874902 14440 raft_consensus.cc:3060] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:01.866529 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 453) took 10378 ms. Trace:
I20251211 08:58:01.879237 14905 rpcz_store.cc:276] 1211 08:57:51.487612 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:51.487720 (+ 108us) service_pool.cc:225] Handling call
1211 08:58:01.860101 (+10372381us) ts_tablet_manager.cc:1151] Deleting tablet da124c4afb5a402fb212d0ea95e0b47e
1211 08:58:01.866212 (+ 6111us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:01.866480 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10371933}
W20251211 08:58:01.880766 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet da124c4afb5a402fb212d0ea95e0b47e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da124c4afb5a402fb212d0ea95e0b47e already in progress: creating tablet
I20251211 08:58:01.881994 14440 raft_consensus.cc:2468] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:01.883008 14542 leader_election.cc:304] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:58:01.883730 14722 tablet_bootstrap.cc:492] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:01.883737 16240 raft_consensus.cc:2804] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:01.884194 14722 ts_tablet_manager.cc:1403] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.193s user 0.012s sys 0.003s
I20251211 08:58:01.884138 16240 raft_consensus.cc:697] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:01.884831 16240 consensus_queue.cc:237] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:01.886586 14722 raft_consensus.cc:359] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.887209 14722 raft_consensus.cc:385] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:01.887473 14722 raft_consensus.cc:740] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
W20251211 08:58:01.887974 16190 log.cc:927] Time spent T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20251211 08:58:01.888068 14722 consensus_queue.cc:260] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.893585 16246 leader_election.cc:290] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:01.899341 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 454) took 9022 ms. Trace:
I20251211 08:58:01.899755 14905 rpcz_store.cc:276] 1211 08:57:52.876789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:52.898295 (+ 21506us) service_pool.cc:225] Handling call
1211 08:58:01.898978 (+9000683us) ts_tablet_manager.cc:1151] Deleting tablet ee50f29bfe674611a834d84a7c4e9070
1211 08:58:01.899021 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:01.899289 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8999376}
W20251211 08:58:01.901024 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet ee50f29bfe674611a834d84a7c4e9070 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee50f29bfe674611a834d84a7c4e9070 already in progress: creating tablet
I20251211 08:58:01.901975 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 455) took 8341 ms. Trace:
I20251211 08:58:01.902287 14905 rpcz_store.cc:276] 1211 08:57:53.560790 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.561067 (+ 277us) service_pool.cc:225] Handling call
1211 08:58:01.901766 (+8340699us) ts_tablet_manager.cc:1151] Deleting tablet 4e813a4c73334c839bfe6af2119e259c
1211 08:58:01.901804 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:01.901931 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8307042}
W20251211 08:58:01.903396 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 4e813a4c73334c839bfe6af2119e259c because the tablet was not found. No further retry: Not found: Tablet not found: 4e813a4c73334c839bfe6af2119e259c
I20251211 08:58:01.904217 14905 tablet_replica.cc:333] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:01.910331 14722 ts_tablet_manager.cc:1434] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20251211 08:58:01.911377 14722 tablet_bootstrap.cc:492] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:01.914664 16062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c9cd45be3947d09423ed4508ca2e91. Latest consensus state: current_term: 2 leader_uuid: "a4c9cd45be3947d09423ed4508ca2e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } } }
I20251211 08:58:01.915313 16062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:01.916220 16087 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4c9cd45be3947d09423ed4508ca2e91)
I20251211 08:58:01.916776 16087 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:58:01.922138 16223 raft_consensus.cc:493] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:01.922545 16223 raft_consensus.cc:515] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.924402 16223 leader_election.cc:290] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:01.925092 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7a8acb51a2948f4a018787bfaecb694" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:01.926290 14385 leader_election.cc:343] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.906467 15030 tablet_bootstrap.cc:492] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:01.927091 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7a8acb51a2948f4a018787bfaecb694" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:01.938956 15030 ts_tablet_manager.cc:1403] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.216s user 0.008s sys 0.001s
W20251211 08:58:01.939980 14382 leader_election.cc:343] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.940366 14382 leader_election.cc:304] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.941465 16223 raft_consensus.cc:2749] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:01.943580 15030 raft_consensus.cc:359] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:01.944156 15030 raft_consensus.cc:385] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:01.944379 15030 raft_consensus.cc:740] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:01.944892 15030 consensus_queue.cc:260] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:01.955606 14904 ts_tablet_manager.cc:1929] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:01.956002 14904 log.cc:1199] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/dd14de369e084828afb8a924c46c649b
I20251211 08:58:01.956565 14904 ts_tablet_manager.cc:1950] T dd14de369e084828afb8a924c46c649b P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:01.958478 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 458) took 6164 ms. Trace:
I20251211 08:58:01.959986 16223 raft_consensus.cc:493] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:01.960494 14905 raft_consensus.cc:2243] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:01.961052 14905 raft_consensus.cc:2272] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:01.960638 16223 raft_consensus.cc:515] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.964094 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd09f7c6364d74bb33503b5bbfea96" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:01.964165 16087 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
W20251211 08:58:01.965378 14385 leader_election.cc:343] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.966197 14904 rpcz_store.cc:276] 1211 08:57:55.793662 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.822307 (+ 28645us) service_pool.cc:225] Handling call
1211 08:58:01.862800 (+6040493us) ts_tablet_manager.cc:1151] Deleting tablet dd14de369e084828afb8a924c46c649b
1211 08:58:01.866183 (+ 3383us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:01.897149 (+ 30966us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:01.955370 (+ 58221us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:01.958415 (+ 3045us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5953926}
I20251211 08:58:01.967090 16223 leader_election.cc:290] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:01.967399 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd09f7c6364d74bb33503b5bbfea96" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:01.967464 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet dd14de369e084828afb8a924c46c649b (table default.Table_b72b9116d4994b90ad29513202ff4217 [id=13c7f04a5e51426f86648847a9fe455c]) successfully deleted
W20251211 08:58:01.968693 14382 leader_election.cc:343] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:01.969100 14382 leader_election.cc:304] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:01.970126 16223 raft_consensus.cc:2749] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:01.973374 14714 tablet_bootstrap.cc:492] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:01.973883 14714 ts_tablet_manager.cc:1403] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.164s user 0.022s sys 0.012s
I20251211 08:58:01.976015 15030 ts_tablet_manager.cc:1434] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20251211 08:58:01.976696 14904 tablet_replica.cc:333] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:01.976897 15030 tablet_bootstrap.cc:492] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:01.977525 14904 raft_consensus.cc:2243] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:01.977972 14904 raft_consensus.cc:2272] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:01.979102 15885 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48320 (request call id 0) took 6822 ms. Trace:
I20251211 08:58:01.962675 14905 ts_tablet_manager.cc:1916] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:01.989436 14714 raft_consensus.cc:359] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.990087 14714 raft_consensus.cc:385] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:01.990338 14714 raft_consensus.cc:740] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
W20251211 08:58:01.991000 16190 log.cc:927] Time spent T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: Append to log took a long time: real 0.056s user 0.003s sys 0.000s
I20251211 08:58:01.993265 14904 ts_tablet_manager.cc:1916] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:01.996454 14714 consensus_queue.cc:260] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:01.998602 16268 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20251211 08:58:02.003108 13902 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 59735.5us
I20251211 08:58:02.011571 16247 raft_consensus.cc:493] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:58:02.012038 16247 raft_consensus.cc:515] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.014760 16247 leader_election.cc:290] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:58:02.015838 14231 leader_election.cc:343] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: c55408f39a414e2caf1db48e6f29eca7
I20251211 08:58:02.016032 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55408f39a414e2caf1db48e6f29eca7" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:02.014839 14714 ts_tablet_manager.cc:1434] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.041s user 0.008s sys 0.001s
I20251211 08:58:02.017901 14230 leader_election.cc:304] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:02.019906 16241 raft_consensus.cc:2749] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:58:02.034799 16240 raft_consensus.cc:493] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:02.035422 16240 raft_consensus.cc:515] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.038795 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e36c6e2868e143be8e7f72307202a268" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:02.040112 14544 leader_election.cc:343] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:02.041296 16240 leader_election.cc:290] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:02.041887 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e36c6e2868e143be8e7f72307202a268" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:02.042441 14440 raft_consensus.cc:2468] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:02.043565 14542 leader_election.cc:304] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:58:02.055318 14904 ts_tablet_manager.cc:1929] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:02.055889 16240 raft_consensus.cc:2804] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:02.061378 16240 raft_consensus.cc:493] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:02.061815 16240 raft_consensus.cc:3060] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:02.062808 14905 ts_tablet_manager.cc:1929] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:02.063156 14905 log.cc:1199] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/da52cace2aab4f9dacef18ec5fa56aaf
I20251211 08:58:02.063751 14905 ts_tablet_manager.cc:1950] T da52cace2aab4f9dacef18ec5fa56aaf P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:02.064103 14904 log.cc:1199] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/1b2445f66be54d3fbdc3be964b9a9132
I20251211 08:58:02.066869 14904 ts_tablet_manager.cc:1950] T 1b2445f66be54d3fbdc3be964b9a9132 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:02.067390 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 65a6feed325f4391a15553e8be914d2d
I20251211 08:58:02.068645 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 458) took 8370 ms. Trace:
I20251211 08:58:02.068866 14260 tablet_service.cc:1552] Processing DeleteTablet for tablet 65a6feed325f4391a15553e8be914d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:59 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:02.068969 14905 rpcz_store.cc:276] 1211 08:57:53.698100 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.705545 (+ 7445us) service_pool.cc:225] Handling call
1211 08:58:01.904086 (+8198541us) ts_tablet_manager.cc:1151] Deleting tablet da52cace2aab4f9dacef18ec5fa56aaf
1211 08:58:01.904122 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.041657 (+137535us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.062579 (+ 20922us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.068581 (+ 6002us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21092,"tablet-delete.queue_time_us":8197903}
I20251211 08:58:02.070398 14004 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55614 (request call id 621) took 2715 ms. Trace:
I20251211 08:58:02.070711 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet da52cace2aab4f9dacef18ec5fa56aaf (table default.Table_5a729546e47b4fc3b53ef93b18b3e438 [id=d41efdbc326e4db681a3750ec6a8d1fe]) successfully deleted
I20251211 08:58:02.071255 14418 tablet_service.cc:1552] Processing DeleteTablet for tablet 65a6feed325f4391a15553e8be914d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:59 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:02.071803 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet 65a6feed325f4391a15553e8be914d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:59 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:02.072675 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 460) took 8220 ms. Trace:
I20251211 08:58:02.072962 14905 rpcz_store.cc:276] 1211 08:57:53.852167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:53.852306 (+ 139us) service_pool.cc:225] Handling call
1211 08:58:02.072337 (+8220031us) ts_tablet_manager.cc:1151] Deleting tablet 07099552b546444bab4d66238dd6d146
1211 08:58:02.072373 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.072635 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8219452}
W20251211 08:58:02.073973 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 07099552b546444bab4d66238dd6d146 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07099552b546444bab4d66238dd6d146 already in progress: creating tablet
I20251211 08:58:02.075065 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 462) took 7910 ms. Trace:
I20251211 08:58:02.075515 14905 rpcz_store.cc:276] 1211 08:57:54.164786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:54.165472 (+ 686us) service_pool.cc:225] Handling call
1211 08:58:02.074602 (+7909130us) ts_tablet_manager.cc:1151] Deleting tablet ca5eebf2ce3848c79ee2fc21ad2b1486
1211 08:58:02.074640 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.075022 (+ 382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7908629}
W20251211 08:58:02.076692 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet ca5eebf2ce3848c79ee2fc21ad2b1486 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca5eebf2ce3848c79ee2fc21ad2b1486 already in progress: creating tablet
I20251211 08:58:02.077813 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 465) took 7252 ms. Trace:
I20251211 08:58:02.078143 14905 rpcz_store.cc:276] 1211 08:57:54.825639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:54.825738 (+ 99us) service_pool.cc:225] Handling call
1211 08:58:02.077457 (+7251719us) ts_tablet_manager.cc:1151] Deleting tablet b49ea29851d94622b4299e93804a0578
1211 08:58:02.077493 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.077758 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7251012}
W20251211 08:58:02.079285 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet b49ea29851d94622b4299e93804a0578 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b49ea29851d94622b4299e93804a0578 already in progress: creating tablet
I20251211 08:58:02.080428 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 467) took 7086 ms. Trace:
I20251211 08:58:02.080765 14905 rpcz_store.cc:276] 1211 08:57:54.994359 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:54.994474 (+ 115us) service_pool.cc:225] Handling call
1211 08:58:02.080147 (+7085673us) ts_tablet_manager.cc:1151] Deleting tablet 143f20f5c37948068ff2fb0d92f0f34b
1211 08:58:02.080184 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.080385 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6962776}
I20251211 08:58:02.081161 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 466) took 5304 ms. Trace:
W20251211 08:58:02.081938 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 143f20f5c37948068ff2fb0d92f0f34b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 143f20f5c37948068ff2fb0d92f0f34b already in progress: creating tablet
I20251211 08:58:02.081550 14904 rpcz_store.cc:276] 1211 08:57:56.776839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:56.787227 (+ 10388us) service_pool.cc:225] Handling call
1211 08:58:01.976535 (+5189308us) ts_tablet_manager.cc:1151] Deleting tablet 1b2445f66be54d3fbdc3be964b9a9132
1211 08:58:01.976578 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.021955 (+ 45377us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.055091 (+ 33136us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.081091 (+ 26000us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5188811}
I20251211 08:58:02.084164 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 1b2445f66be54d3fbdc3be964b9a9132 (table default.Table_e71106e05fbc47248f2b5f71482c2231 [id=2cc6cd83f4414e91b8359c3b65336061]) successfully deleted
I20251211 08:58:02.084365 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 472) took 6617 ms. Trace:
I20251211 08:58:02.084983 14905 rpcz_store.cc:276] 1211 08:57:55.467433 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.467551 (+ 118us) service_pool.cc:225] Handling call
1211 08:58:02.082649 (+6615098us) ts_tablet_manager.cc:1151] Deleting tablet f2dc6cb4efc744e6904728ad120df934
1211 08:58:02.082683 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.084322 (+ 1639us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6611560}
W20251211 08:58:02.086134 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet f2dc6cb4efc744e6904728ad120df934 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2dc6cb4efc744e6904728ad120df934 already in progress: creating tablet
I20251211 08:58:02.087085 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 473) took 6481 ms. Trace:
I20251211 08:58:02.087393 14905 rpcz_store.cc:276] 1211 08:57:55.605737 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.605865 (+ 128us) service_pool.cc:225] Handling call
1211 08:58:02.086817 (+6480952us) ts_tablet_manager.cc:1151] Deleting tablet 14fc87cb76024b5b8227dfe9939e1c89
1211 08:58:02.086853 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.087043 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6480397}
I20251211 08:58:02.095290 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet a2530df4933249e19d96759910a47673
I20251211 08:58:02.096271 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 596 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:58:02.097854 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet a2530df4933249e19d96759910a47673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:58:00 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:02.097927 14009 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55614 (request call id 633) took 1766 ms. Trace:
I20251211 08:58:02.098665 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 467) took 5299 ms. Trace:
I20251211 08:58:02.098976 14904 rpcz_store.cc:276] 1211 08:57:56.798867 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:56.798971 (+ 104us) service_pool.cc:225] Handling call
1211 08:58:02.082622 (+5283651us) ts_tablet_manager.cc:1151] Deleting tablet 225438e70aae424a9671aaceb3ee0ed6
1211 08:58:02.096661 (+ 14039us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.098612 (+ 1951us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5283043}
I20251211 08:58:02.099435 14260 tablet_service.cc:1552] Processing DeleteTablet for tablet a2530df4933249e19d96759910a47673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:58:00 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:02.100076 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 479) took 4551 ms. Trace:
I20251211 08:58:02.100355 14904 rpcz_store.cc:276] 1211 08:57:57.548450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.548547 (+ 97us) service_pool.cc:225] Handling call
1211 08:58:02.099766 (+4551219us) ts_tablet_manager.cc:1151] Deleting tablet 476eee1606e74832abf928dbaafc844d
1211 08:58:02.099805 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.100027 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4435249}
W20251211 08:58:02.100483 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 225438e70aae424a9671aaceb3ee0ed6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 225438e70aae424a9671aaceb3ee0ed6 already in progress: creating tablet
I20251211 08:58:02.101336 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 486) took 4536 ms. Trace:
I20251211 08:58:02.101615 14904 rpcz_store.cc:276] 1211 08:57:57.564512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.570208 (+ 5696us) service_pool.cc:225] Handling call
1211 08:58:02.101112 (+4530904us) ts_tablet_manager.cc:1151] Deleting tablet 5f7775994f684326b6312c802f6b084d
1211 08:58:02.101149 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.101295 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4434085}
W20251211 08:58:02.101907 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 476eee1606e74832abf928dbaafc844d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 476eee1606e74832abf928dbaafc844d already in progress: creating tablet
I20251211 08:58:02.097973 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet a2530df4933249e19d96759910a47673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:58:00 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:02.103065 16246 raft_consensus.cc:493] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
W20251211 08:58:02.104705 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 5f7775994f684326b6312c802f6b084d because the tablet was not found. No further retry: Not found: Tablet not found: 5f7775994f684326b6312c802f6b084d
I20251211 08:58:02.104838 16246 raft_consensus.cc:515] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:58:02.105439 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 14fc87cb76024b5b8227dfe9939e1c89 because the tablet was not found. No further retry: Not found: Tablet not found: 14fc87cb76024b5b8227dfe9939e1c89
I20251211 08:58:02.106033 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 474) took 6498 ms. Trace:
I20251211 08:58:02.098222 14009 rpcz_store.cc:276] 1211 08:58:00.331458 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.397907 (+ 66449us) service_pool.cc:225] Handling call
1211 08:58:00.399137 (+ 1230us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:58:00.399701 (+ 564us) catalog_manager.cc:2815] Modifying in-memory table state
1211 08:58:00.399848 (+ 147us) catalog_manager.cc:2824] Locking tablets
1211 08:58:02.060050 (+1660202us) catalog_manager.cc:2838] Removing table and tablets from system table
1211 08:58:02.094885 (+ 34835us) catalog_manager.cc:2856] Removing table from by-name map
1211 08:58:02.095084 (+ 199us) catalog_manager.cc:2872] Committing in-memory state
1211 08:58:02.095182 (+ 98us) catalog_manager.cc:2876] Aborting table tasks
1211 08:58:02.097877 (+ 2695us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:02.062403 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:02.062798 (+ 395us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:02.062819 (+ 21us) write_op.cc:435] Acquired schema lock
1211 08:58:02.062832 (+ 13us) tablet.cc:660] Decoding operations
1211 08:58:02.063128 (+ 296us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:02.063162 (+ 34us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:02.063184 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:02.063422 (+ 238us) tablet.cc:699] Row locks acquired
1211 08:58:02.063436 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:58:02.063546 (+ 110us) write_op.cc:270] Start()
1211 08:58:02.063619 (+ 73us) write_op.cc:276] Timestamp: P: 1765443482063505 usec, L: 0
1211 08:58:02.063634 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:58:02.064134 (+ 500us) log.cc:844] Serialized 759 byte log entry
1211 08:58:02.083365 (+ 19231us) op_driver.cc:464] REPLICATION: finished
1211 08:58:02.088836 (+ 5471us) write_op.cc:301] APPLY: starting
1211 08:58:02.088919 (+ 83us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:02.089096 (+ 177us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:02.089226 (+ 130us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:02.089894 (+ 668us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:02.090121 (+ 227us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:58:02.090201 (+ 80us) write_op.cc:312] APPLY: finished
1211 08:58:02.090605 (+ 404us) log.cc:844] Serialized 29 byte log entry
1211 08:58:02.090872 (+ 267us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:02.090965 (+ 93us) write_op.cc:454] Released schema lock
1211 08:58:02.091012 (+ 47us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5230,"num_ops":2,"prepare.queue_time_us":719,"prepare.run_cpu_time_us":1910,"prepare.run_wall_time_us":1908,"replication_time_us":19618,"spinlock_wait_cycles":90240}]]}
I20251211 08:58:02.106407 14905 rpcz_store.cc:276] 1211 08:57:55.607554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.607938 (+ 384us) service_pool.cc:225] Handling call
1211 08:58:02.105737 (+6497799us) ts_tablet_manager.cc:1151] Deleting tablet 00e5c4bf71b44abca66b0e94995c533e
1211 08:58:02.105776 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.105985 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6497315}
W20251211 08:58:02.108155 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 00e5c4bf71b44abca66b0e94995c533e because the tablet was not found. No further retry: Not found: Tablet not found: 00e5c4bf71b44abca66b0e94995c533e
W20251211 08:58:02.108474 14542 leader_election.cc:343] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: c55408f39a414e2caf1db48e6f29eca7
I20251211 08:58:02.109175 16246 leader_election.cc:290] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:58:02.109822 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_184dbe4f30324cfcbb04f5031169242e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:02.070791 14004 rpcz_store.cc:276] 1211 08:57:59.354891 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.416631 (+ 61740us) service_pool.cc:225] Handling call
1211 08:57:59.651824 (+235193us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:57:59.652342 (+ 518us) catalog_manager.cc:2815] Modifying in-memory table state
1211 08:57:59.652480 (+ 138us) catalog_manager.cc:2824] Locking tablets
1211 08:58:02.046734 (+2394254us) catalog_manager.cc:2838] Removing table and tablets from system table
1211 08:58:02.066942 (+ 20208us) catalog_manager.cc:2856] Removing table from by-name map
1211 08:58:02.067109 (+ 167us) catalog_manager.cc:2872] Committing in-memory state
1211 08:58:02.067197 (+ 88us) catalog_manager.cc:2876] Aborting table tasks
1211 08:58:02.070348 (+ 3151us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:02.054918 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:02.055303 (+ 385us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:02.055325 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:58:02.055336 (+ 11us) tablet.cc:660] Decoding operations
1211 08:58:02.055726 (+ 390us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:02.055768 (+ 42us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:02.055789 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:02.056066 (+ 277us) tablet.cc:699] Row locks acquired
1211 08:58:02.056078 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:58:02.056210 (+ 132us) write_op.cc:270] Start()
1211 08:58:02.056293 (+ 83us) write_op.cc:276] Timestamp: P: 1765443482056142 usec, L: 0
1211 08:58:02.056309 (+ 16us) op_driver.cc:348] REPLICATION: starting
1211 08:58:02.056826 (+ 517us) log.cc:844] Serialized 759 byte log entry
1211 08:58:02.064403 (+ 7577us) op_driver.cc:464] REPLICATION: finished
1211 08:58:02.065350 (+ 947us) write_op.cc:301] APPLY: starting
1211 08:58:02.065398 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:02.065496 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:02.065635 (+ 139us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:02.065930 (+ 295us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:02.066052 (+ 122us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:58:02.066093 (+ 41us) write_op.cc:312] APPLY: finished
1211 08:58:02.066257 (+ 164us) log.cc:844] Serialized 29 byte log entry
1211 08:58:02.066385 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:02.066441 (+ 56us) write_op.cc:454] Released schema lock
1211 08:58:02.066468 (+ 27us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:02.066661 (+ 193us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:02.066685 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":805,"apply.run_cpu_time_us":1287,"apply.run_wall_time_us":1287,"num_ops":2,"prepare.queue_time_us":6058,"prepare.run_cpu_time_us":2219,"prepare.run_wall_time_us":7392,"raft.queue_time_us":94,"raft.run_cpu_time_us":788,"raft.run_wall_time_us":786,"replication_time_us":8018}]]}
W20251211 08:58:02.111768 14009 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_184dbe4f30324cfcbb04f5031169242e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:02.107327 16240 raft_consensus.cc:515] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.113695 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_1b2a75580e22420a81aa5a88d27de913"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:02.109184 14905 tablet_replica.cc:333] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:02.115089 14905 raft_consensus.cc:2243] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:02.115989 14905 raft_consensus.cc:2272] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:02.116158 14714 tablet_bootstrap.cc:492] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:02.121212 14714 tablet_bootstrap.cc:654] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:02.125938 15030 tablet_bootstrap.cc:654] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:58:02.126699 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 65a6feed325f4391a15553e8be914d2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65a6feed325f4391a15553e8be914d2d already in progress: creating tablet
W20251211 08:58:02.127346 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet a2530df4933249e19d96759910a47673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2530df4933249e19d96759910a47673 already in progress: creating tablet
I20251211 08:58:02.129480 16219 consensus_queue.cc:1048] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:58:02.133872 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e36c6e2868e143be8e7f72307202a268" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:02.134439 14440 raft_consensus.cc:3060] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:02.102560 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 487) took 4537 ms. Trace:
I20251211 08:58:02.135383 14904 rpcz_store.cc:276] 1211 08:57:57.564865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.565054 (+ 189us) service_pool.cc:225] Handling call
1211 08:58:02.102356 (+4537302us) ts_tablet_manager.cc:1151] Deleting tablet 45c7d841bd3643a799a8b96e82d6d675
1211 08:58:02.102389 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.102518 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4428553}
W20251211 08:58:02.136509 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 45c7d841bd3643a799a8b96e82d6d675 because the tablet was not found. No further retry: Not found: Tablet not found: 45c7d841bd3643a799a8b96e82d6d675
I20251211 08:58:02.137645 14904 tablet_replica.cc:333] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:02.138330 14904 raft_consensus.cc:2243] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:02.109685 16247 raft_consensus.cc:493] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:02.138809 14904 raft_consensus.cc:2272] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:02.139019 16247 raft_consensus.cc:515] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.140587 14904 ts_tablet_manager.cc:1916] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:02.140799 16247 leader_election.cc:290] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:02.111020 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55408f39a414e2caf1db48e6f29eca7" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:02.141579 14299 raft_consensus.cc:2468] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:02.142643 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c02cb6ec01041dba118683a57aecde2" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:02.143214 14449 raft_consensus.cc:2468] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:58:02.114960 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e36c6e2868e143be8e7f72307202a268" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:02.143491 14010 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:58:02.144567 14231 leader_election.cc:304] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:58:02.145913 16247 raft_consensus.cc:2804] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:02.146135 14010 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:02.146255 16247 raft_consensus.cc:493] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:02.146625 16247 raft_consensus.cc:3060] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:02.146870 14010 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_3528b4ed9d9742dda51c5fd712140bd7 [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 3 got 0
I20251211 08:58:02.147794 14010 catalog_manager.cc:5654] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20251211 08:58:02.149127 14010 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:02.160256 14722 tablet_bootstrap.cc:654] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
W20251211 08:58:02.165087 13998 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 48575fa3f45349b9887bfc1cdfebc958 (table default.Table_a7ed3d0a6ce0464893bc9f5bb3bf6851 [id=1b3efd8519a04b13af86aa631f596f33], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 48575fa3f45349b9887bfc1cdfebc958 consensus state has no leader
I20251211 08:58:02.178961 14010 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:02.182580 14418 tablet_service.cc:1505] Processing CreateTablet for tablet 55b0870999a34ce3ac1daaaa488b68dc (DEFAULT_TABLE table=default.Table_40b8667dac894419bfd51474ff42c299 [id=b7e399f1891548078d04f863e639f2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.183790 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55b0870999a34ce3ac1daaaa488b68dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.134897 16240 leader_election.cc:290] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:02.147472 14544 leader_election.cc:304] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:58:02.156248 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 55b0870999a34ce3ac1daaaa488b68dc (DEFAULT_TABLE table=default.Table_40b8667dac894419bfd51474ff42c299 [id=b7e399f1891548078d04f863e639f2fb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:58:02.196015 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 48670.8us
W20251211 08:58:02.196951 14544 leader_election.cc:343] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:02.198222 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c02cb6ec01041dba118683a57aecde2" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:02.199316 14440 raft_consensus.cc:2468] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:02.200006 16240 raft_consensus.cc:2804] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:58:02.200609 16240 raft_consensus.cc:493] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:58:02.200942 14542 leader_election.cc:304] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:58:02.201152 16240 raft_consensus.cc:3060] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:58:02.201623 16246 raft_consensus.cc:2804] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:02.203984 14260 tablet_service.cc:1505] Processing CreateTablet for tablet 55b0870999a34ce3ac1daaaa488b68dc (DEFAULT_TABLE table=default.Table_40b8667dac894419bfd51474ff42c299 [id=b7e399f1891548078d04f863e639f2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.204820 14427 tablet_service.cc:1505] Processing CreateTablet for tablet c4d6e5abc2a04830b3d6cedb5144f65d (DEFAULT_TABLE table=default.Table_5467e261c9f5488f9084d51d89721df3 [id=381296faa1de4a40b2ea5db390ea1bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.206069 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d6e5abc2a04830b3d6cedb5144f65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.206362 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55b0870999a34ce3ac1daaaa488b68dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.206236 14574 tablet_service.cc:1505] Processing CreateTablet for tablet c4d6e5abc2a04830b3d6cedb5144f65d (DEFAULT_TABLE table=default.Table_5467e261c9f5488f9084d51d89721df3 [id=381296faa1de4a40b2ea5db390ea1bee]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:58:02.208491 14230 leader_election.cc:343] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:02.208571 16246 raft_consensus.cc:697] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:02.210770 14266 tablet_service.cc:1505] Processing CreateTablet for tablet c4d6e5abc2a04830b3d6cedb5144f65d (DEFAULT_TABLE table=default.Table_5467e261c9f5488f9084d51d89721df3 [id=381296faa1de4a40b2ea5db390ea1bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.211678 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d6e5abc2a04830b3d6cedb5144f65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.209496 16246 consensus_queue.cc:237] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.221992 16219 consensus_queue.cc:1048] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:02.232980 14419 tablet_service.cc:1505] Processing CreateTablet for tablet 3d3ffe3fef0e4c1592ac18ad52716a8e (DEFAULT_TABLE table=default.Table_2aa82bb39d744859b7dc4e25a5c495f4 [id=5279e23a06e84291adb987163858db6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.234163 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3ffe3fef0e4c1592ac18ad52716a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.243273 14714 tablet_bootstrap.cc:492] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:02.243697 14714 ts_tablet_manager.cc:1403] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.128s user 0.011s sys 0.004s
W20251211 08:58:02.247306 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b2a75580e22420a81aa5a88d27de913 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:02.250722 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 596 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:58:02.251626 14905 ts_tablet_manager.cc:1916] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:58:02.267215 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 58790.5us
I20251211 08:58:02.271543 14714 raft_consensus.cc:359] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.272213 14714 raft_consensus.cc:385] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:02.272423 14714 raft_consensus.cc:740] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:02.273022 14714 consensus_queue.cc:260] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.275514 16223 raft_consensus.cc:493] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:02.275929 16223 raft_consensus.cc:515] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.278692 16223 leader_election.cc:290] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:02.286679 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55b0870999a34ce3ac1daaaa488b68dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.288705 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "476eee1606e74832abf928dbaafc844d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:02.291873 14385 leader_election.cc:343] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:02.295656 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:02.297452 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (table default.Table_321194b0f71d457e81e14c810650b486 [id=f74e074787144a21ad31d4d89cc509c9]). Expected version 1 got 0
I20251211 08:58:02.298696 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_31fcfbed4a2c4e8aa05fb42c261bf1d5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 2 got 0
I20251211 08:58:02.299284 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_3528b4ed9d9742dda51c5fd712140bd7 [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 3 got 0
I20251211 08:58:02.300020 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet f5c5b1aebde94dea81cb37a6a6998070 (table default.Table_cd86506c20bc42359393525a923146bd [id=615b65833c3446ca94f57fbcf1659333]). Expected version 1 got 0
I20251211 08:58:02.302134 14714 ts_tablet_manager.cc:1434] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.058s user 0.004s sys 0.002s
I20251211 08:58:02.307103 15726 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:58:02.312014 16096 consensus_queue.cc:1048] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251211 08:58:02.300072 16247 raft_consensus.cc:515] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.318501 16247 leader_election.cc:290] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:02.320194 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c02cb6ec01041dba118683a57aecde2" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:02.320729 14449 raft_consensus.cc:3060] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:02.327354 14413 tablet_service.cc:1505] Processing CreateTablet for tablet eaf7a9b771c5418eae0cd620fdff3ae3 (DEFAULT_TABLE table=default.Table_dfdb7a58dd4c42b9a73c75fd44f74fd6 [id=8ebc400f8dc440a99636c0a8d213890c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.328424 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf7a9b771c5418eae0cd620fdff3ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.329926 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 3d3ffe3fef0e4c1592ac18ad52716a8e (DEFAULT_TABLE table=default.Table_2aa82bb39d744859b7dc4e25a5c495f4 [id=5279e23a06e84291adb987163858db6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.331517 14581 tablet_service.cc:1505] Processing CreateTablet for tablet 3d3ffe3fef0e4c1592ac18ad52716a8e (DEFAULT_TABLE table=default.Table_2aa82bb39d744859b7dc4e25a5c495f4 [id=5279e23a06e84291adb987163858db6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.332448 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3ffe3fef0e4c1592ac18ad52716a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.334457 14605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c02cb6ec01041dba118683a57aecde2" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:58:02.335127 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "476eee1606e74832abf928dbaafc844d" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:02.328435 14449 raft_consensus.cc:2468] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:58:02.339715 14905 ts_tablet_manager.cc:1929] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:02.340039 14905 log.cc:1199] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/2a1b011cb7c647f79bb16ce1c92bcff5
I20251211 08:58:02.339862 14231 leader_election.cc:304] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:58:02.340703 14905 ts_tablet_manager.cc:1950] T 2a1b011cb7c647f79bb16ce1c92bcff5 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:02.340770 16241 consensus_queue.cc:1048] T 0fdc020f21aa4f35880a435d7cfb605d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:02.341888 16241 raft_consensus.cc:2804] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:02.342240 16241 raft_consensus.cc:697] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:58:02.342473 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 475) took 6734 ms. Trace:
I20251211 08:58:02.342727 14905 rpcz_store.cc:276] 1211 08:57:55.607797 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:55.608038 (+ 241us) service_pool.cc:225] Handling call
1211 08:58:02.108973 (+6500935us) ts_tablet_manager.cc:1151] Deleting tablet 2a1b011cb7c647f79bb16ce1c92bcff5
1211 08:58:02.109086 (+ 113us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.320455 (+211369us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.339509 (+ 19054us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.342422 (+ 2913us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6500543}
I20251211 08:58:02.343021 16241 consensus_queue.cc:237] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:02.343943 14382 leader_election.cc:343] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:02.344322 14382 leader_election.cc:304] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:02.344774 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 477) took 5795 ms. Trace:
I20251211 08:58:02.345232 16285 raft_consensus.cc:2749] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:02.345042 14905 rpcz_store.cc:276] 1211 08:57:56.549515 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:56.549637 (+ 122us) service_pool.cc:225] Handling call
1211 08:58:02.344468 (+5794831us) ts_tablet_manager.cc:1151] Deleting tablet 785a5edbda984467b897ee1bbaf1430c
1211 08:58:02.344508 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.344722 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5793355}
I20251211 08:58:02.346230 14905 tablet_replica.cc:333] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:02.347275 14722 tablet_bootstrap.cc:492] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:02.347903 14905 raft_consensus.cc:2243] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:02.348619 14905 pending_rounds.cc:70] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:58:02.349371 14905 pending_rounds.cc:77] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256500992815104 op_type: NO_OP noop_request { }
I20251211 08:58:02.349740 14905 raft_consensus.cc:2889] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:58:02.350096 14905 raft_consensus.cc:2272] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:02.351810 14574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d6e5abc2a04830b3d6cedb5144f65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.354955 14905 ts_tablet_manager.cc:1916] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:02.363301 14722 ts_tablet_manager.cc:1403] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.452s user 0.013s sys 0.006s
I20251211 08:58:02.365624 14722 raft_consensus.cc:359] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.366254 14722 raft_consensus.cc:385] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:02.367153 16087 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251211 08:58:02.367405 15726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.60:38671, user_credentials={real_user=slave}} blocked reactor thread for 65636.5us
I20251211 08:58:02.371531 15849 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b276f77f87254a5d909c5ab940a39256" instance_seqno: 1765443440621960) as {username='slave'} at 127.13.146.2:46747; Asking this server to re-register.
W20251211 08:58:02.380375 14230 leader_election.cc:343] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:02.332775 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_b258afac6a8e48bcacfc67b25ed79e41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:02.382172 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b258afac6a8e48bcacfc67b25ed79e41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:02.384013 14904 ts_tablet_manager.cc:1929] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:02.384377 14904 log.cc:1199] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/e9905227c0a844d199362a442e7d630c
I20251211 08:58:02.343952 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 2a1b011cb7c647f79bb16ce1c92bcff5 (table default.Table_03bee763242749cfa5fb0fe7d7e7fb7f [id=209008ecbdd4441989221f0f956a87b6]) successfully deleted
I20251211 08:58:02.385076 14904 ts_tablet_manager.cc:1950] T e9905227c0a844d199362a442e7d630c P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:58:02.385457 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 785a5edbda984467b897ee1bbaf1430c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 785a5edbda984467b897ee1bbaf1430c already in progress: creating tablet
I20251211 08:58:02.387602 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:02.366560 14722 raft_consensus.cc:740] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:02.388967 14722 consensus_queue.cc:260] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.391253 14722 ts_tablet_manager.cc:1434] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20251211 08:58:02.384974 16240 raft_consensus.cc:515] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.396425 14581 tablet_service.cc:1505] Processing CreateTablet for tablet eaf7a9b771c5418eae0cd620fdff3ae3 (DEFAULT_TABLE table=default.Table_dfdb7a58dd4c42b9a73c75fd44f74fd6 [id=8ebc400f8dc440a99636c0a8d213890c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.398089 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf7a9b771c5418eae0cd620fdff3ae3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:58:02.398288 14542 leader_election.cc:343] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: c55408f39a414e2caf1db48e6f29eca7
I20251211 08:58:02.400883 15030 tablet_bootstrap.cc:492] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:02.401531 15030 ts_tablet_manager.cc:1403] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.425s user 0.009s sys 0.005s
I20251211 08:58:02.401595 14497 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:02.316727 14714 tablet_bootstrap.cc:492] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:02.403656 15030 raft_consensus.cc:359] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.404311 15030 raft_consensus.cc:385] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:02.404580 15030 raft_consensus.cc:740] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:02.409165 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55408f39a414e2caf1db48e6f29eca7" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:02.409704 14299 raft_consensus.cc:3060] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:58:02.413652 15849 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" instance_seqno: 1765443443098816) as {username='slave'} at 127.13.146.3:44575; Asking this server to re-register.
I20251211 08:58:02.396625 14004 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7d8b607b4ed9468ab5b3f58936dd0d7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:02.416391 14299 raft_consensus.cc:2468] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 2.
W20251211 08:58:02.416502 14004 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d8b607b4ed9468ab5b3f58936dd0d7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:02.417682 14544 leader_election.cc:304] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:58:02.425938 16246 raft_consensus.cc:2804] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader election won for term 2
I20251211 08:58:02.426748 16246 raft_consensus.cc:697] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:02.427630 16246 consensus_queue.cc:237] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.407935 16240 leader_election.cc:290] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 election: Requested vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:58:02.437568 14004 master_service.cc:954] ReplaceTablet: received request to replace tablet 033c2c016f8f4c01adc8bdbf2885e394 from {username='slave'} at 127.0.0.1:55614
I20251211 08:58:02.438416 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:02.439196 14011 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:58:02.441875 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3ffe3fef0e4c1592ac18ad52716a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.447430 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 488) took 4864 ms. Trace:
I20251211 08:58:02.447754 14904 rpcz_store.cc:276] 1211 08:57:57.582596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.582757 (+ 161us) service_pool.cc:225] Handling call
1211 08:58:02.137433 (+4554676us) ts_tablet_manager.cc:1151] Deleting tablet e9905227c0a844d199362a442e7d630c
1211 08:58:02.137515 (+ 82us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.264562 (+127047us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.383804 (+119242us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.447376 (+ 63572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4459781}
I20251211 08:58:02.448788 14904 tablet_replica.cc:333] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:02.450296 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet e9905227c0a844d199362a442e7d630c (table default.Table_edfe3b698dfa4a7b9cd5a1d18521d00b [id=45f67c67257e428f8ed2f16898ad5fe6]) successfully deleted
I20251211 08:58:02.450462 14714 tablet_bootstrap.cc:654] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:02.451848 14722 tablet_bootstrap.cc:492] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:02.456905 14260 tablet_service.cc:1505] Processing CreateTablet for tablet eaf7a9b771c5418eae0cd620fdff3ae3 (DEFAULT_TABLE table=default.Table_dfdb7a58dd4c42b9a73c75fd44f74fd6 [id=8ebc400f8dc440a99636c0a8d213890c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.458012 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf7a9b771c5418eae0cd620fdff3ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.459578 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 6b6fa6ef0cad464aacaefcf775927229 (DEFAULT_TABLE table=default.Table_f8b62284383b447f88f4ecaa25393a1f [id=c49c06d10bf3471f993c6d95f33d1136]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.460493 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b6fa6ef0cad464aacaefcf775927229. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.461325 14418 tablet_service.cc:1505] Processing CreateTablet for tablet e1a07744ceb34f4ea48d235e335490ff (DEFAULT_TABLE table=default.Table_61cd87d7f57741dd997ee9bac6835add [id=566678eae64b4188851f743e3dfb1cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.465533 16247 raft_consensus.cc:493] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:02.465934 16247 raft_consensus.cc:515] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.476420 16247 leader_election.cc:290] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:02.477532 14413 tablet_service.cc:1505] Processing CreateTablet for tablet 6b6fa6ef0cad464aacaefcf775927229 (DEFAULT_TABLE table=default.Table_f8b62284383b447f88f4ecaa25393a1f [id=c49c06d10bf3471f993c6d95f33d1136]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:58:02.477885 14231 leader_election.cc:343] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 00e5c4bf71b44abca66b0e94995c533e
I20251211 08:58:02.479365 15030 consensus_queue.cc:260] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.481351 14722 tablet_bootstrap.cc:654] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:02.481663 15030 ts_tablet_manager.cc:1434] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.080s user 0.004s sys 0.000s
I20251211 08:58:02.481896 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1a07744ceb34f4ea48d235e335490ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.482412 15030 tablet_bootstrap.cc:492] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:02.485939 14655 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:02.486619 14267 tablet_service.cc:1505] Processing CreateTablet for tablet e1a07744ceb34f4ea48d235e335490ff (DEFAULT_TABLE table=default.Table_61cd87d7f57741dd997ee9bac6835add [id=566678eae64b4188851f743e3dfb1cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.487522 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1a07744ceb34f4ea48d235e335490ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.487969 14574 tablet_service.cc:1505] Processing CreateTablet for tablet e1a07744ceb34f4ea48d235e335490ff (DEFAULT_TABLE table=default.Table_61cd87d7f57741dd997ee9bac6835add [id=566678eae64b4188851f743e3dfb1cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.489209 14574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1a07744ceb34f4ea48d235e335490ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.491827 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_3528b4ed9d9742dda51c5fd712140bd7 [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 3 got 2
I20251211 08:58:02.493014 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:02.493491 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 6b6fa6ef0cad464aacaefcf775927229 (DEFAULT_TABLE table=default.Table_f8b62284383b447f88f4ecaa25393a1f [id=c49c06d10bf3471f993c6d95f33d1136]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.494342 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b6fa6ef0cad464aacaefcf775927229. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.493953 14012 catalog_manager.cc:5654] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to b276f77f87254a5d909c5ab940a39256 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "b276f77f87254a5d909c5ab940a39256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } }
I20251211 08:58:02.495257 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:02.499368 14904 raft_consensus.cc:2243] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:02.499945 14904 raft_consensus.cc:2272] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:02.501880 14904 ts_tablet_manager.cc:1916] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:02.421816 14905 ts_tablet_manager.cc:1929] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:02.502923 14905 log.cc:1199] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/1b2445f66be54d3fbdc3be964b9a9132
I20251211 08:58:02.504680 14905 ts_tablet_manager.cc:1950] T 1b2445f66be54d3fbdc3be964b9a9132 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:02.495609 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b6fa6ef0cad464aacaefcf775927229. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.506526 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 481) took 5727 ms. Trace:
I20251211 08:58:02.519214 16285 raft_consensus.cc:493] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:02.519698 16285 raft_consensus.cc:515] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.521436 15847 ts_manager.cc:194] Registered new tserver with Master: b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:02.522159 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07f983604bfa474f9c9ce0c982d738fc" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:02.523370 14385 leader_election.cc:343] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:02.524412 16285 leader_election.cc:290] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:02.531985 14905 rpcz_store.cc:276] 1211 08:57:56.779286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:56.779369 (+ 83us) service_pool.cc:225] Handling call
1211 08:58:02.346100 (+5566731us) ts_tablet_manager.cc:1151] Deleting tablet 1b2445f66be54d3fbdc3be964b9a9132
1211 08:58:02.346136 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.364467 (+ 18331us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.421617 (+ 57150us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.506468 (+ 84851us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5566298}
I20251211 08:58:02.535220 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 1b2445f66be54d3fbdc3be964b9a9132 (table default.Table_e71106e05fbc47248f2b5f71482c2231 [id=2cc6cd83f4414e91b8359c3b65336061]) successfully deleted
I20251211 08:58:02.536185 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 482) took 5737 ms. Trace:
I20251211 08:58:02.536787 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07f983604bfa474f9c9ce0c982d738fc" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:02.536942 16241 consensus_queue.cc:1048] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:02.536543 14905 rpcz_store.cc:276] 1211 08:57:56.798336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:56.798428 (+ 92us) service_pool.cc:225] Handling call
1211 08:58:02.535888 (+5737460us) ts_tablet_manager.cc:1151] Deleting tablet 225438e70aae424a9671aaceb3ee0ed6
1211 08:58:02.535926 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.536137 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5736978}
I20251211 08:58:02.538228 15030 tablet_bootstrap.cc:654] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:58:02.538380 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 225438e70aae424a9671aaceb3ee0ed6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 225438e70aae424a9671aaceb3ee0ed6 already in progress: creating tablet
I20251211 08:58:02.540710 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 494) took 5347 ms. Trace:
I20251211 08:58:02.541054 14905 rpcz_store.cc:276] 1211 08:57:57.193003 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.193174 (+ 171us) service_pool.cc:225] Handling call
1211 08:58:02.539193 (+5346019us) ts_tablet_manager.cc:1151] Deleting tablet 476eee1606e74832abf928dbaafc844d
1211 08:58:02.540389 (+ 1196us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.540660 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5345405}
W20251211 08:58:02.542207 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 476eee1606e74832abf928dbaafc844d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 476eee1606e74832abf928dbaafc844d already in progress: creating tablet
W20251211 08:58:02.552596 14382 leader_election.cc:343] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:02.552968 14382 leader_election.cc:304] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:02.556254 16285 raft_consensus.cc:2749] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:02.559032 14714 tablet_bootstrap.cc:492] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:02.559517 14714 ts_tablet_manager.cc:1403] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.243s user 0.006s sys 0.007s
I20251211 08:58:02.559646 15030 tablet_bootstrap.cc:492] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:02.560021 15030 ts_tablet_manager.cc:1403] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.007s
I20251211 08:58:02.561941 14714 raft_consensus.cc:359] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.562049 15030 raft_consensus.cc:359] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.562556 15030 raft_consensus.cc:385] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:02.562834 15030 raft_consensus.cc:740] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:02.563062 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 501) took 5080 ms. Trace:
I20251211 08:58:02.563367 14905 rpcz_store.cc:276] 1211 08:57:57.482582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.482846 (+ 264us) service_pool.cc:225] Handling call
1211 08:58:02.541871 (+5059025us) ts_tablet_manager.cc:1151] Deleting tablet 1a2dcb9ac62e4aac885d638f4e2b072e
1211 08:58:02.562847 (+ 20976us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.563011 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4914868}
I20251211 08:58:02.563375 15030 consensus_queue.cc:260] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:58:02.564749 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 1a2dcb9ac62e4aac885d638f4e2b072e because the tablet was not found. No further retry: Not found: Tablet not found: 1a2dcb9ac62e4aac885d638f4e2b072e
W20251211 08:58:02.565752 14230 leader_election.cc:343] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: 00e5c4bf71b44abca66b0e94995c533e
I20251211 08:58:02.566238 15030 ts_tablet_manager.cc:1434] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20251211 08:58:02.566226 14230 leader_election.cc:304] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:02.567376 16247 raft_consensus.cc:2749] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:02.570566 15030 tablet_bootstrap.cc:492] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:02.568459 14714 raft_consensus.cc:385] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:02.575165 14418 tablet_service.cc:1505] Processing CreateTablet for tablet ab8f3f5fef0a4b12ac5c741ca442af56 (DEFAULT_TABLE table=default.Table_b96aa9120021443caa9cf1d91e71e6b6 [id=ed9dbabdb4ec4c8d9d9d279304ed764b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.576164 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab8f3f5fef0a4b12ac5c741ca442af56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.576195 14574 tablet_service.cc:1505] Processing CreateTablet for tablet ab8f3f5fef0a4b12ac5c741ca442af56 (DEFAULT_TABLE table=default.Table_b96aa9120021443caa9cf1d91e71e6b6 [id=ed9dbabdb4ec4c8d9d9d279304ed764b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.577342 14574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab8f3f5fef0a4b12ac5c741ca442af56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.582837 14714 raft_consensus.cc:740] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:02.583575 14714 consensus_queue.cc:260] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.587852 15849 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "31404d5f58e548519239569b150d975b" instance_seqno: 1765443438143206) as {username='slave'} at 127.13.146.1:35431; Asking this server to re-register.
I20251211 08:58:02.589182 14266 tablet_service.cc:1505] Processing CreateTablet for tablet ab8f3f5fef0a4b12ac5c741ca442af56 (DEFAULT_TABLE table=default.Table_b96aa9120021443caa9cf1d91e71e6b6 [id=ed9dbabdb4ec4c8d9d9d279304ed764b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.591751 14349 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:02.591897 14419 tablet_service.cc:1505] Processing CreateTablet for tablet 5e56780077374c16ae3e1b035dc0fc3d (DEFAULT_TABLE table=default.Table_56e0be458512461a81a0341c520b616d [id=ba0964c2b2054d7eaeb8df3612662ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.592486 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 5e56780077374c16ae3e1b035dc0fc3d (DEFAULT_TABLE table=default.Table_56e0be458512461a81a0341c520b616d [id=ba0964c2b2054d7eaeb8df3612662ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.592929 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e56780077374c16ae3e1b035dc0fc3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.593541 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e56780077374c16ae3e1b035dc0fc3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.599813 14722 tablet_bootstrap.cc:492] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:02.600236 14722 ts_tablet_manager.cc:1403] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.149s user 0.009s sys 0.001s
I20251211 08:58:02.606257 14005 catalog_manager.cc:5654] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20251211 08:58:02.608294 14905 tablet_replica.cc:333] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:02.609064 14905 raft_consensus.cc:2243] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:02.609570 14905 raft_consensus.cc:2272] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:02.611794 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 5e56780077374c16ae3e1b035dc0fc3d (DEFAULT_TABLE table=default.Table_56e0be458512461a81a0341c520b616d [id=ba0964c2b2054d7eaeb8df3612662ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.616750 15847 ts_manager.cc:194] Registered new tserver with Master: c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:02.622262 15030 tablet_bootstrap.cc:654] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:02.622815 14714 ts_tablet_manager.cc:1434] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.063s user 0.003s sys 0.001s
I20251211 08:58:02.623071 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:02.623725 14714 tablet_bootstrap.cc:492] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:02.602368 14722 raft_consensus.cc:359] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.627063 14722 raft_consensus.cc:385] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:02.627349 14722 raft_consensus.cc:740] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:02.628015 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:02.627902 14722 consensus_queue.cc:260] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.629891 14722 ts_tablet_manager.cc:1434] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20251211 08:58:02.635444 14413 tablet_service.cc:1505] Processing CreateTablet for tablet 7ac030260df9467bace1d759b03e3cdb (DEFAULT_TABLE table=default.Table_a2712ccf332e4a14a29f57badb7591cc [id=bd3f42250cb1402ca549203279d0ffc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.636380 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ac030260df9467bace1d759b03e3cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.635773 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:58:02.641121 14263 tablet_service.cc:1505] Processing CreateTablet for tablet 7ac030260df9467bace1d759b03e3cdb (DEFAULT_TABLE table=default.Table_a2712ccf332e4a14a29f57badb7591cc [id=bd3f42250cb1402ca549203279d0ffc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.641561 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:02.643577 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab8f3f5fef0a4b12ac5c741ca442af56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.647202 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e56780077374c16ae3e1b035dc0fc3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.659488 14263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ac030260df9467bace1d759b03e3cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.659489 14722 tablet_bootstrap.cc:492] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:02.633675 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 7ac030260df9467bace1d759b03e3cdb (DEFAULT_TABLE table=default.Table_a2712ccf332e4a14a29f57badb7591cc [id=bd3f42250cb1402ca549203279d0ffc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.663784 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ac030260df9467bace1d759b03e3cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.646586 14905 ts_tablet_manager.cc:1916] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:02.670430 14714 tablet_bootstrap.cc:654] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:02.694900 16285 raft_consensus.cc:493] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:02.695453 16285 raft_consensus.cc:515] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.697224 16285 leader_election.cc:290] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:02.698823 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7af7c9ab27a64a1681f9874a922e986f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:02.700048 14385 leader_election.cc:343] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:02.711616 14904 ts_tablet_manager.cc:1929] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:02.712008 14904 log.cc:1199] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/c341b762afd8440bb7198ca134c8dffe
I20251211 08:58:02.712709 14904 ts_tablet_manager.cc:1950] T c341b762afd8440bb7198ca134c8dffe P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:02.725521 14714 tablet_bootstrap.cc:492] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:02.726008 14714 ts_tablet_manager.cc:1403] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.102s user 0.013s sys 0.004s
I20251211 08:58:02.734041 14714 raft_consensus.cc:359] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.740842 14714 raft_consensus.cc:385] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:02.741140 14714 raft_consensus.cc:740] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:02.744036 14714 consensus_queue.cc:260] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.747774 14714 ts_tablet_manager.cc:1434] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.021s user 0.003s sys 0.005s
I20251211 08:58:02.749732 14413 tablet_service.cc:1505] Processing CreateTablet for tablet f2eb23700a1243a5bceda42f8d6f2c2b (DEFAULT_TABLE table=default.Table_b931f8327b1c4aea804749feb7a41f6e [id=87c231a92de04101a6ced58b7a7b01f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.753783 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 489) took 4883 ms. Trace:
I20251211 08:58:02.756846 14714 tablet_bootstrap.cc:492] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:02.756865 14722 tablet_bootstrap.cc:654] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:02.757313 14586 tablet_service.cc:1505] Processing CreateTablet for tablet f2eb23700a1243a5bceda42f8d6f2c2b (DEFAULT_TABLE table=default.Table_b931f8327b1c4aea804749feb7a41f6e [id=87c231a92de04101a6ced58b7a7b01f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.758544 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2eb23700a1243a5bceda42f8d6f2c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.759709 14267 tablet_service.cc:1505] Processing CreateTablet for tablet f2eb23700a1243a5bceda42f8d6f2c2b (DEFAULT_TABLE table=default.Table_b931f8327b1c4aea804749feb7a41f6e [id=87c231a92de04101a6ced58b7a7b01f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.761698 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2eb23700a1243a5bceda42f8d6f2c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.764694 14418 tablet_service.cc:1505] Processing CreateTablet for tablet 6ed40d17fa72458a963c5ada637b61cc (DEFAULT_TABLE table=default.Table_6d15b83379564d4cbcf8329733fded8b [id=0f7da3b742474beeabc5986b039ede3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.765537 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ed40d17fa72458a963c5ada637b61cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.767402 14577 tablet_service.cc:1505] Processing CreateTablet for tablet 6ed40d17fa72458a963c5ada637b61cc (DEFAULT_TABLE table=default.Table_6d15b83379564d4cbcf8329733fded8b [id=0f7da3b742474beeabc5986b039ede3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.773772 14260 tablet_service.cc:1505] Processing CreateTablet for tablet 6ed40d17fa72458a963c5ada637b61cc (DEFAULT_TABLE table=default.Table_6d15b83379564d4cbcf8329733fded8b [id=0f7da3b742474beeabc5986b039ede3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.778611 16247 raft_consensus.cc:493] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:02.779098 16247 raft_consensus.cc:515] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:02.782742 14231 leader_election.cc:343] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 51c3f5eaef784e868e565f81bd51a5e4
I20251211 08:58:02.783036 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
I20251211 08:58:02.783496 16247 leader_election.cc:290] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:02.784389 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet d7a8acb51a2948f4a018787bfaecb694 (table default.Table_9e007a9469d54d99b9471cb8e653b620 [id=de3a242b7e6d480ebf509e458ec002c7]). Expected version 1 got 0
I20251211 08:58:02.785089 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:02.785981 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:02.786330 16246 raft_consensus.cc:493] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:02.786780 16246 raft_consensus.cc:515] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.787475 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ed40d17fa72458a963c5ada637b61cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.788506 16246 leader_election.cc:290] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:02.791057 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cc3373962574253868ababbbe78e57d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:02.791612 14440 raft_consensus.cc:2468] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:02.792471 14542 leader_election.cc:304] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:02.793097 16246 raft_consensus.cc:2804] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:02.793380 16246 raft_consensus.cc:493] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:02.793648 16246 raft_consensus.cc:3060] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:02.794061 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2eb23700a1243a5bceda42f8d6f2c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.804425 16246 raft_consensus.cc:515] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.807145 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cc3373962574253868ababbbe78e57d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:02.807758 14440 raft_consensus.cc:3060] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:02.807940 16240 raft_consensus.cc:493] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:02.808339 16240 raft_consensus.cc:515] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.811443 16240 leader_election.cc:290] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:02.812420 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51c3f5eaef784e868e565f81bd51a5e4" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:02.812812 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac6be204692a44fab2db551e79237299" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:02.813040 14599 raft_consensus.cc:2468] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:58:02.813381 14441 raft_consensus.cc:2468] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:02.814440 14542 leader_election.cc:304] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:02.815121 14599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7af7c9ab27a64a1681f9874a922e986f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:02.815248 16240 raft_consensus.cc:2804] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:02.816103 16240 raft_consensus.cc:493] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:02.816494 16240 raft_consensus.cc:3060] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:02.824349 14440 raft_consensus.cc:2468] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:02.825253 14542 leader_election.cc:304] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:02.825969 15902 raft_consensus.cc:2804] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:02.826928 14419 tablet_service.cc:1505] Processing CreateTablet for tablet a8b666397a5d4cac9fec1949f47e42c4 (DEFAULT_TABLE table=default.Table_07dddc710b174f5ca438600d9fe40b91 [id=c847a55379bd4fb99557df214b499910]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.827857 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8b666397a5d4cac9fec1949f47e42c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.832988 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cc3373962574253868ababbbe78e57d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:02.833514 14299 raft_consensus.cc:2468] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:02.835232 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cc3373962574253868ababbbe78e57d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:02.835749 14299 raft_consensus.cc:3060] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:02.754118 14904 rpcz_store.cc:276] 1211 08:57:57.870373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.920689 (+ 50316us) service_pool.cc:225] Handling call
1211 08:58:02.448641 (+4527952us) ts_tablet_manager.cc:1151] Deleting tablet c341b762afd8440bb7198ca134c8dffe
1211 08:58:02.448679 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.588283 (+139604us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.654079 (+ 65796us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:02.753719 (+ 99640us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49615,"tablet-delete.queue_time_us":4527387}
I20251211 08:58:02.839712 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet c341b762afd8440bb7198ca134c8dffe (table default.Table_96d05b744df34836be68b4841e6c2a2b [id=86d8260b23ca4945acd9a70054b77c26]) successfully deleted
I20251211 08:58:02.841797 14299 raft_consensus.cc:2468] T 9cc3373962574253868ababbbe78e57d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:02.843641 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 490) took 4896 ms. Trace:
I20251211 08:58:02.843942 14904 rpcz_store.cc:276] 1211 08:57:57.947673 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.947809 (+ 136us) service_pool.cc:225] Handling call
1211 08:58:02.843417 (+4895608us) ts_tablet_manager.cc:1151] Deleting tablet ccce5a446ac9460c9c9b7de72d875d95
1211 08:58:02.843456 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.843593 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4895038}
I20251211 08:58:02.846964 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ed40d17fa72458a963c5ada637b61cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.847047 14006 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:02.852589 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac6be204692a44fab2db551e79237299" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:02.853286 14297 raft_consensus.cc:2468] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:02.855047 14230 leader_election.cc:304] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:58:02.855672 16247 consensus_queue.cc:1048] T 7c0bdd2d0d9f4221bf13812bb7a97025 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:02.856243 16096 raft_consensus.cc:2804] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:58:02.856560 16096 raft_consensus.cc:493] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:02.856985 16096 raft_consensus.cc:3060] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:58:02.863886 16096 raft_consensus.cc:515] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:02.856559 14230 consensus_peers.cc:597] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:02.866649 16096 leader_election.cc:290] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:02.867604 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 491) took 4838 ms. Trace:
I20251211 08:58:02.867933 14904 rpcz_store.cc:276] 1211 08:57:58.029131 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.034323 (+ 5192us) service_pool.cc:225] Handling call
1211 08:58:02.867296 (+4832973us) ts_tablet_manager.cc:1151] Deleting tablet 15595c5a432047de873c2647b78838d6
1211 08:58:02.867336 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.867543 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4832195}
I20251211 08:58:02.868958 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 492) took 4815 ms. Trace:
I20251211 08:58:02.869242 14904 rpcz_store.cc:276] 1211 08:57:58.054479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.054692 (+ 213us) service_pool.cc:225] Handling call
1211 08:58:02.868681 (+4813989us) ts_tablet_manager.cc:1151] Deleting tablet 2c0f8d7dd90f4beaae674fe827d37e0b
1211 08:58:02.868726 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.868915 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4813074}
W20251211 08:58:02.871285 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet ccce5a446ac9460c9c9b7de72d875d95 because the tablet was not found. No further retry: Not found: Tablet not found: ccce5a446ac9460c9c9b7de72d875d95
W20251211 08:58:02.872231 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 15595c5a432047de873c2647b78838d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15595c5a432047de873c2647b78838d6 already in progress: creating tablet
W20251211 08:58:02.872927 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 2c0f8d7dd90f4beaae674fe827d37e0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c0f8d7dd90f4beaae674fe827d37e0b already in progress: creating tablet
W20251211 08:58:02.878536 14382 leader_election.cc:343] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:02.878968 14382 leader_election.cc:304] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:02.881388 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51c3f5eaef784e868e565f81bd51a5e4" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:58:02.881914 14598 raft_consensus.cc:3060] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:58:02.897924 15849 ts_manager.cc:194] Registered new tserver with Master: 31404d5f58e548519239569b150d975b (127.13.146.1:36181)
I20251211 08:58:02.905027 14722 tablet_bootstrap.cc:492] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:02.905570 14722 ts_tablet_manager.cc:1403] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.246s user 0.009s sys 0.004s
I20251211 08:58:02.910560 14418 tablet_service.cc:1505] Processing CreateTablet for tablet 4054b3b38a5144e1a6af165000a5946d (DEFAULT_TABLE table=default.Table_6229702ff48d405ab70d847e8418262a [id=124269cd63254dcf8ef373fb3f0d7f1b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:58:02.910894 14231 leader_election.cc:343] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 51c3f5eaef784e868e565f81bd51a5e4
I20251211 08:58:02.911725 14418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4054b3b38a5144e1a6af165000a5946d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.914902 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:58:02.916429 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:02.917492 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:02.918272 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
I20251211 08:58:02.933774 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 493) took 4862 ms. Trace:
I20251211 08:58:02.934175 14904 rpcz_store.cc:276] 1211 08:57:58.071404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.105200 (+ 33796us) service_pool.cc:225] Handling call
1211 08:58:02.869973 (+4764773us) ts_tablet_manager.cc:1151] Deleting tablet eccc727462b447489372da09ea828ac2
1211 08:58:02.933440 (+ 63467us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.933716 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4721342}
I20251211 08:58:02.935357 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 496) took 4404 ms. Trace:
W20251211 08:58:02.935492 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet eccc727462b447489372da09ea828ac2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eccc727462b447489372da09ea828ac2 already in progress: creating tablet
I20251211 08:58:02.935635 14904 rpcz_store.cc:276] 1211 08:57:58.531314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.531428 (+ 114us) service_pool.cc:225] Handling call
1211 08:58:02.935046 (+4403618us) ts_tablet_manager.cc:1151] Deleting tablet 020230ff08ed42179a28efbb21613a0f
1211 08:58:02.935091 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.935309 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4346662}
I20251211 08:58:02.936722 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 497) took 4374 ms. Trace:
W20251211 08:58:02.936818 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 020230ff08ed42179a28efbb21613a0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 020230ff08ed42179a28efbb21613a0f already in progress: creating tablet
I20251211 08:58:02.936977 14904 rpcz_store.cc:276] 1211 08:57:58.562626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.562720 (+ 94us) service_pool.cc:225] Handling call
1211 08:58:02.936448 (+4373728us) ts_tablet_manager.cc:1151] Deleting tablet 3aed0129e9164a779b73994b8b37426c
1211 08:58:02.936484 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.936681 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4346780}
W20251211 08:58:02.937955 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 3aed0129e9164a779b73994b8b37426c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aed0129e9164a779b73994b8b37426c already in progress: creating tablet
I20251211 08:58:02.948666 16219 raft_consensus.cc:2749] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:02.952107 14263 tablet_service.cc:1505] Processing CreateTablet for tablet a8b666397a5d4cac9fec1949f47e42c4 (DEFAULT_TABLE table=default.Table_07dddc710b174f5ca438600d9fe40b91 [id=c847a55379bd4fb99557df214b499910]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.955499 14714 tablet_bootstrap.cc:654] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:02.960667 14722 raft_consensus.cc:359] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:02.961525 14722 raft_consensus.cc:385] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:02.961835 14722 raft_consensus.cc:740] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:02.964078 14009 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:02.967494 16246 leader_election.cc:290] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:02.976702 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 4054b3b38a5144e1a6af165000a5946d (DEFAULT_TABLE table=default.Table_6229702ff48d405ab70d847e8418262a [id=124269cd63254dcf8ef373fb3f0d7f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:02.822533 16240 raft_consensus.cc:515] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.982326 15902 raft_consensus.cc:697] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:02.983521 15902 consensus_queue.cc:237] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:02.980823 16240 leader_election.cc:290] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:03.002133 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 960e6bf66ad84af3a68ac9047c4d7f83 (DEFAULT_TABLE table=default.Table_0275161db5ae45b6950bd8b95723b20b [id=e1e1d62a8477419d9139c9606f23be0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.006419 15030 tablet_bootstrap.cc:492] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:03.006897 15030 ts_tablet_manager.cc:1403] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.437s user 0.018s sys 0.004s
I20251211 08:58:03.011076 16285 raft_consensus.cc:493] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:03.011572 16285 raft_consensus.cc:515] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.013520 16285 leader_election.cc:290] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:03.014271 14598 raft_consensus.cc:2468] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 2.
I20251211 08:58:03.021790 14414 tablet_service.cc:1505] Processing CreateTablet for tablet 716295d94c884a3c9cae77200b611878 (DEFAULT_TABLE table=default.Table_cd8592260266488598378cde91871ad9 [id=ff13003e3a294a2e8acbae3bd9d645d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.031971 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 716295d94c884a3c9cae77200b611878 (DEFAULT_TABLE table=default.Table_cd8592260266488598378cde91871ad9 [id=ff13003e3a294a2e8acbae3bd9d645d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.036787 15030 raft_consensus.cc:359] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.037379 15030 raft_consensus.cc:385] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:03.037592 15030 raft_consensus.cc:740] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:03.037956 14415 tablet_service.cc:1505] Processing CreateTablet for tablet baf86be1945043a6bf5e8e7f4ecb8ede (DEFAULT_TABLE table=default.Table_38faf33a40be4a6fa97f912efdc62b1b [id=53bb113fbaa04d469ef4bed9fffa9ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.038365 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac6be204692a44fab2db551e79237299" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:03.038152 15030 consensus_queue.cc:260] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.038852 14449 raft_consensus.cc:3060] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:03.049078 14449 raft_consensus.cc:2468] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:03.052739 14905 ts_tablet_manager.cc:1929] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:03.053107 14905 log.cc:1199] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/f3403116517f45c59f87fa5bda0b4071
W20251211 08:58:03.053346 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 174923us
I20251211 08:58:03.053759 14905 ts_tablet_manager.cc:1950] T f3403116517f45c59f87fa5bda0b4071 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:03.062404 14004 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 033c2c016f8f4c01adc8bdbf2885e394
I20251211 08:58:03.065514 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 960e6bf66ad84af3a68ac9047c4d7f83 (DEFAULT_TABLE table=default.Table_0275161db5ae45b6950bd8b95723b20b [id=e1e1d62a8477419d9139c9606f23be0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.069738 16311 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4c9cd45be3947d09423ed4508ca2e91)
I20251211 08:58:03.071692 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 502) took 5588 ms. Trace:
I20251211 08:58:03.072187 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet 033c2c016f8f4c01adc8bdbf2885e394 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f0bb5daf5c04698bad02fb5fa2e987e) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:03.071959 14905 rpcz_store.cc:276] 1211 08:57:57.483377 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.483877 (+ 500us) service_pool.cc:225] Handling call
1211 08:58:02.608083 (+5124206us) ts_tablet_manager.cc:1151] Deleting tablet f3403116517f45c59f87fa5bda0b4071
1211 08:58:02.608121 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:02.941174 (+333053us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:03.052507 (+111333us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:03.071637 (+ 19130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4980445}
I20251211 08:58:03.073050 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 503) took 5584 ms. Trace:
I20251211 08:58:03.073316 14905 rpcz_store.cc:276] 1211 08:57:57.488383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.488491 (+ 108us) service_pool.cc:225] Handling call
1211 08:58:03.072736 (+5584245us) ts_tablet_manager.cc:1151] Deleting tablet 15595c5a432047de873c2647b78838d6
1211 08:58:03.072772 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.072993 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5444401}
I20251211 08:58:03.074095 14004 catalog_manager.cc:6724] ReplaceTablet: tablet 033c2c016f8f4c01adc8bdbf2885e394 deleted and replaced by tablet 6f0bb5daf5c04698bad02fb5fa2e987e
I20251211 08:58:03.074312 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 504) took 5498 ms. Trace:
I20251211 08:58:03.074717 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet 033c2c016f8f4c01adc8bdbf2885e394 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f0bb5daf5c04698bad02fb5fa2e987e) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:03.077030 14278 tablet_service.cc:1505] Processing CreateTablet for tablet baf86be1945043a6bf5e8e7f4ecb8ede (DEFAULT_TABLE table=default.Table_38faf33a40be4a6fa97f912efdc62b1b [id=53bb113fbaa04d469ef4bed9fffa9ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.080350 14004 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55614 (request call id 654) took 1254 ms. Trace:
I20251211 08:58:03.081133 14004 rpcz_store.cc:276] 1211 08:58:01.826215 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:02.437438 (+611223us) service_pool.cc:225] Handling call
1211 08:58:03.080295 (+642857us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:03.043782 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:03.044098 (+ 316us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:03.044118 (+ 20us) write_op.cc:435] Acquired schema lock
1211 08:58:03.044128 (+ 10us) tablet.cc:660] Decoding operations
1211 08:58:03.044368 (+ 240us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:03.044401 (+ 33us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:03.044420 (+ 19us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:03.044563 (+ 143us) tablet.cc:699] Row locks acquired
1211 08:58:03.044574 (+ 11us) write_op.cc:260] PREPARE: finished
1211 08:58:03.044658 (+ 84us) write_op.cc:270] Start()
1211 08:58:03.044721 (+ 63us) write_op.cc:276] Timestamp: P: 1765443483044824 usec, L: 0
1211 08:58:03.044734 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:58:03.045115 (+ 381us) log.cc:844] Serialized 659 byte log entry
1211 08:58:03.052773 (+ 7658us) op_driver.cc:464] REPLICATION: finished
1211 08:58:03.053099 (+ 326us) write_op.cc:301] APPLY: starting
1211 08:58:03.053151 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:03.053248 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:03.053351 (+ 103us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:03.053696 (+ 345us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:03.053871 (+ 175us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:03.053910 (+ 39us) write_op.cc:312] APPLY: finished
1211 08:58:03.054085 (+ 175us) log.cc:844] Serialized 29 byte log entry
1211 08:58:03.054202 (+ 117us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:03.054267 (+ 65us) write_op.cc:454] Released schema lock
1211 08:58:03.054293 (+ 26us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:03.054471 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:03.054492 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":196,"apply.run_cpu_time_us":1351,"apply.run_wall_time_us":1350,"num_ops":2,"prepare.queue_time_us":80,"prepare.run_cpu_time_us":1599,"prepare.run_wall_time_us":1598,"raft.queue_time_us":161,"raft.run_cpu_time_us":522,"raft.run_wall_time_us":521,"replication_time_us":7955}]]}
I20251211 08:58:03.074568 14905 rpcz_store.cc:276] 1211 08:57:57.575829 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.575933 (+ 104us) service_pool.cc:225] Handling call
1211 08:58:03.074026 (+5498093us) ts_tablet_manager.cc:1151] Deleting tablet 2c0f8d7dd90f4beaae674fe827d37e0b
1211 08:58:03.074061 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.074264 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38,"tablet-delete.queue_time_us":5445653}
I20251211 08:58:03.088138 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 505) took 5291 ms. Trace:
I20251211 08:58:03.088150 14425 tablet_service.cc:1505] Processing CreateTablet for tablet de6c58a36e5e4f15b41cbe8d14d30a22 (DEFAULT_TABLE table=default.Table_f7dd35033be14c598ae0f9801daa1c4b [id=3e5bd5e839aa4b3fb67fa798c2be3b95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.088395 14905 rpcz_store.cc:276] 1211 08:57:57.796291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:57.798844 (+ 2553us) service_pool.cc:225] Handling call
1211 08:58:03.087826 (+5288982us) ts_tablet_manager.cc:1151] Deleting tablet eccc727462b447489372da09ea828ac2
1211 08:58:03.087861 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.088081 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5288449}
I20251211 08:58:03.089687 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 508) took 4566 ms. Trace:
I20251211 08:58:03.089588 14268 tablet_service.cc:1505] Processing CreateTablet for tablet de6c58a36e5e4f15b41cbe8d14d30a22 (DEFAULT_TABLE table=default.Table_f7dd35033be14c598ae0f9801daa1c4b [id=3e5bd5e839aa4b3fb67fa798c2be3b95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.090063 14905 rpcz_store.cc:276] 1211 08:57:58.523533 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.523665 (+ 132us) service_pool.cc:225] Handling call
1211 08:58:03.089306 (+4565641us) ts_tablet_manager.cc:1151] Deleting tablet 020230ff08ed42179a28efbb21613a0f
1211 08:58:03.089351 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.089626 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4565049}
I20251211 08:58:03.091404 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 509) took 4529 ms. Trace:
I20251211 08:58:03.091743 14905 rpcz_store.cc:276] 1211 08:57:58.562333 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.562461 (+ 128us) service_pool.cc:225] Handling call
1211 08:58:03.091033 (+4528572us) ts_tablet_manager.cc:1151] Deleting tablet 3aed0129e9164a779b73994b8b37426c
1211 08:58:03.091080 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.091336 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4486480}
I20251211 08:58:03.092711 14905 tablet_replica.cc:333] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:03.093561 14905 raft_consensus.cc:2243] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: Raft consensus shutting down.
I20251211 08:58:03.094349 14905 pending_rounds.cc:70] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: Trying to abort 1 pending ops.
I20251211 08:58:03.105173 14414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716295d94c884a3c9cae77200b611878. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.070464 16311 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:58:03.106617 15030 ts_tablet_manager.cc:1434] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.099s user 0.006s sys 0.000s
I20251211 08:58:03.107930 15030 tablet_bootstrap.cc:492] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:03.108825 14415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baf86be1945043a6bf5e8e7f4ecb8ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.113822 14103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 596 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:58:03.116043 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5884dc6ef814552893cdf9dccfcce19" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:03.116410 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de6c58a36e5e4f15b41cbe8d14d30a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.094591 14905 pending_rounds.cc:77] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7231256504034848768 op_type: NO_OP noop_request { }
I20251211 08:58:03.119005 14905 raft_consensus.cc:2889] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:58:03.119339 14905 raft_consensus.cc:2272] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:03.120100 15030 tablet_bootstrap.cc:654] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:03.121551 14905 ts_tablet_manager.cc:1916] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:03.122068 14263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8b666397a5d4cac9fec1949f47e42c4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:58:03.122822 14385 leader_election.cc:343] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:03.124974 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960e6bf66ad84af3a68ac9047c4d7f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.127010 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4054b3b38a5144e1a6af165000a5946d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:02.962530 14722 consensus_queue.cc:260] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.130054 14722 ts_tablet_manager.cc:1434] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.224s user 0.006s sys 0.000s
I20251211 08:58:03.131016 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960e6bf66ad84af3a68ac9047c4d7f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.134629 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baf86be1945043a6bf5e8e7f4ecb8ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.139112 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716295d94c884a3c9cae77200b611878. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.140152 16320 tablet_replica.cc:333] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:03.140964 16320 raft_consensus.cc:2243] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:03.141438 16320 raft_consensus.cc:2272] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:03.109328 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 596 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:58:03.148301 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de6c58a36e5e4f15b41cbe8d14d30a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.137630 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:03.157159 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 58d06c7007354844bdaaf3d26871b5ab (DEFAULT_TABLE table=default.Table_507f5535a8db4b5ba1c1d46b38a612ae [id=dc5b8bcdd735479eace6c7d56adc139b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.158460 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d06c7007354844bdaaf3d26871b5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.159447 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 58d06c7007354844bdaaf3d26871b5ab (DEFAULT_TABLE table=default.Table_507f5535a8db4b5ba1c1d46b38a612ae [id=dc5b8bcdd735479eace6c7d56adc139b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.160470 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d06c7007354844bdaaf3d26871b5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.175720 16307 raft_consensus.cc:493] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:03.175905 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 88860fa9d7e6414185f0a1cbc78b2571 (table default.Table_3528b4ed9d9742dda51c5fd712140bd7 [id=ffc60f5888b54383922a7334cc661ec0]). Expected version 3 got 2
I20251211 08:58:03.176303 16307 raft_consensus.cc:515] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.177237 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:03.177839 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:03.178077 16307 leader_election.cc:290] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:03.178527 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:58:03.180583 14722 tablet_bootstrap.cc:492] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:03.187507 16321 tablet.cc:1722] T 17ab48721a5c48a7b8a5550be658837d P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:58:03.190106 16320 ts_tablet_manager.cc:1916] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:03.194579 14413 tablet_service.cc:1505] Processing CreateTablet for tablet 6cecac6803124fb3a1ae254bd4b35225 (DEFAULT_TABLE table=default.Table_6856babaa0d64d45902ac1bf4c4f8b0e [id=caca4123579b4f91b7f8f2a046ecd999]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.195513 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cecac6803124fb3a1ae254bd4b35225. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.202082 14904 tablet_replica.cc:333] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:03.202832 14904 raft_consensus.cc:2243] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:03.203298 14904 raft_consensus.cc:2272] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:03.205022 14904 ts_tablet_manager.cc:1916] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:03.209640 14714 tablet_bootstrap.cc:492] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:03.210068 14714 ts_tablet_manager.cc:1403] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.453s user 0.006s sys 0.004s
I20251211 08:58:03.211791 16322 tablet.cc:1722] T 17ab48721a5c48a7b8a5550be658837d P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:58:03.216536 14007 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55614 (request call id 640) took 2362 ms. Trace:
I20251211 08:58:03.223161 14260 tablet_service.cc:1505] Processing CreateTablet for tablet 52ae270f69724ef4b11f8831906add57 (DEFAULT_TABLE table=default.Table_2a5d1e8d612c4fc5954ea3efc00eb00b [id=ad68b61e866c40f28e5dbd143ccfcb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.216830 14007 rpcz_store.cc:276] 1211 08:58:00.854194 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.857581 (+ 3387us) service_pool.cc:225] Handling call
1211 08:58:00.910353 (+ 52772us) spinlock_profiling.cc:243] Waited 52.1 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da87016 00007fc20b6b5b24 00007fc20b6b5ab2 00007fc20b6b59e1 00007fc20b6b595c 00007fc20b6b4c12 00007fc20998455c 00007fc209983996
1211 08:58:00.910931 (+ 578us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:58:00.911549 (+ 618us) catalog_manager.cc:3708] Apply alter schema
1211 08:58:00.912034 (+ 485us) catalog_manager.cc:3740] Acquired catalog manager lock
1211 08:58:00.912217 (+ 183us) catalog_manager.cc:3898] Updating metadata on disk
1211 08:58:01.073532 (+161315us) catalog_manager.cc:3929] Committing alterations to in-memory state
1211 08:58:03.216479 (+2142947us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:00.921796 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:00.922110 (+ 314us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:00.922131 (+ 21us) write_op.cc:435] Acquired schema lock
1211 08:58:00.922142 (+ 11us) tablet.cc:660] Decoding operations
1211 08:58:00.922367 (+ 225us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:00.922397 (+ 30us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:00.922416 (+ 19us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:00.922521 (+ 105us) tablet.cc:699] Row locks acquired
1211 08:58:00.922533 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:58:00.922614 (+ 81us) write_op.cc:270] Start()
1211 08:58:00.922674 (+ 60us) write_op.cc:276] Timestamp: P: 1765443480922747 usec, L: 0
1211 08:58:00.922688 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:58:00.923106 (+ 418us) log.cc:844] Serialized 405 byte log entry
1211 08:58:01.050694 (+127588us) op_driver.cc:464] REPLICATION: finished
1211 08:58:01.054273 (+ 3579us) write_op.cc:301] APPLY: starting
1211 08:58:01.054328 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:01.054420 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:01.054527 (+ 107us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:01.054739 (+ 212us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:01.054908 (+ 169us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:01.054963 (+ 55us) write_op.cc:312] APPLY: finished
1211 08:58:01.055141 (+ 178us) log.cc:844] Serialized 23 byte log entry
1211 08:58:01.055776 (+ 635us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:01.055860 (+ 84us) write_op.cc:454] Released schema lock
1211 08:58:01.055915 (+ 55us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":119883008,"child_traces":[["op",{"apply.queue_time_us":3048,"apply.run_cpu_time_us":1352,"apply.run_wall_time_us":2202,"num_ops":1,"prepare.queue_time_us":8628,"prepare.run_cpu_time_us":1659,"prepare.run_wall_time_us":115886,"raft.queue_time_us":90,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":212,"replication_time_us":127921}]]}
I20251211 08:58:03.227660 14419 tablet_service.cc:1505] Processing CreateTablet for tablet 5a6cefecce45444e8e76f5a66154adbd (DEFAULT_TABLE table=default.Table_c58caf908ea44ee18089745fb0a5c6da [id=441c4878bcb64a53adaabf692e1f512e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.228711 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a6cefecce45444e8e76f5a66154adbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.228698 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet f3403116517f45c59f87fa5bda0b4071 (table default.Table_6109f16f9985461fbb07b1d083bbb166 [id=58f79d91f6e64d7980b96715d3dd25e7]) successfully deleted
W20251211 08:58:03.229748 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 15595c5a432047de873c2647b78838d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15595c5a432047de873c2647b78838d6 already in progress: creating tablet
W20251211 08:58:03.230836 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 2c0f8d7dd90f4beaae674fe827d37e0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c0f8d7dd90f4beaae674fe827d37e0b already in progress: creating tablet
W20251211 08:58:03.231828 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet eccc727462b447489372da09ea828ac2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eccc727462b447489372da09ea828ac2 already in progress: creating tablet
W20251211 08:58:03.234004 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 020230ff08ed42179a28efbb21613a0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 020230ff08ed42179a28efbb21613a0f already in progress: creating tablet
W20251211 08:58:03.234588 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 3aed0129e9164a779b73994b8b37426c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3aed0129e9164a779b73994b8b37426c already in progress: creating tablet
I20251211 08:58:03.235421 14268 tablet_service.cc:1505] Processing CreateTablet for tablet 6cecac6803124fb3a1ae254bd4b35225 (DEFAULT_TABLE table=default.Table_6856babaa0d64d45902ac1bf4c4f8b0e [id=caca4123579b4f91b7f8f2a046ecd999]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.236635 14542 leader_election.cc:304] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:03.237757 14714 raft_consensus.cc:359] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.238584 14714 raft_consensus.cc:385] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:03.238886 14714 raft_consensus.cc:740] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:03.239528 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ae270f69724ef4b11f8831906add57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.239588 14714 consensus_queue.cc:260] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.247714 14012 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_48a06c2ab96d42feb95c8813fb635c80 [id=26c0a9d9b1f84372b18a54f045eb768c] alter complete (version 6)
I20251211 08:58:03.248718 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cecac6803124fb3a1ae254bd4b35225. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.263064 14230 leader_election.cc:304] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:58:03.265501 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825d8cf806d94702bdd11e0d63ee0c59" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:03.265697 16096 raft_consensus.cc:2804] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Leader election won for term 2
I20251211 08:58:03.266180 16096 raft_consensus.cc:697] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 2 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:58:03.266978 16096 consensus_queue.cc:237] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:03.268465 14385 leader_election.cc:343] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:03.269547 14714 ts_tablet_manager.cc:1434] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.059s user 0.004s sys 0.000s
I20251211 08:58:03.269907 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac6be204692a44fab2db551e79237299" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:03.270422 14299 raft_consensus.cc:3060] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:03.271550 15758 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:58:03.276080 14006 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_3528b4ed9d9742dda51c5fd712140bd7 [id=ffc60f5888b54383922a7334cc661ec0] alter complete (version 3)
I20251211 08:58:03.272493 14415 tablet_service.cc:1552] Processing DeleteTablet for tablet ed0e90efe651499c929b1fb992712b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:03.280256 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c02cb6ec01041dba118683a57aecde2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:03.281137 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 570c7f08c46e4273bd38c32593dcf304 with delete_type TABLET_DATA_DELETED (replaced by tablet 3838fd220d034b3595f351b906188339) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:03.283146 14421 tablet_service.cc:1552] Processing DeleteTablet for tablet c112dfcc301648cda6f0f038b1f2af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:03.282544 14299 raft_consensus.cc:2468] T ac6be204692a44fab2db551e79237299 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:03.292057 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 5a6cefecce45444e8e76f5a66154adbd (DEFAULT_TABLE table=default.Table_c58caf908ea44ee18089745fb0a5c6da [id=441c4878bcb64a53adaabf692e1f512e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.292959 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a6cefecce45444e8e76f5a66154adbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.269687 16311 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:58:03.299252 16311 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251211 08:58:03.301872 14421 tablet_service.cc:1505] Processing CreateTablet for tablet df06100dbe4e43fc9840a8cc959d47bf (DEFAULT_TABLE table=default.Table_cc7fd4312d5d4dc7807e350123a637e0 [id=ae4b8a68a7f2484fa0980dae0d574de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.301721 14573 tablet_service.cc:1505] Processing CreateTablet for tablet 5a6cefecce45444e8e76f5a66154adbd (DEFAULT_TABLE table=default.Table_c58caf908ea44ee18089745fb0a5c6da [id=441c4878bcb64a53adaabf692e1f512e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.302788 14421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df06100dbe4e43fc9840a8cc959d47bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.303280 14278 tablet_service.cc:1505] Processing CreateTablet for tablet df06100dbe4e43fc9840a8cc959d47bf (DEFAULT_TABLE table=default.Table_cc7fd4312d5d4dc7807e350123a637e0 [id=ae4b8a68a7f2484fa0980dae0d574de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.304163 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df06100dbe4e43fc9840a8cc959d47bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.304785 14573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a6cefecce45444e8e76f5a66154adbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.314038 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet ac6be204692a44fab2db551e79237299 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c0f8d7dd90f4beaae674fe827d37e0b) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:03.315222 14575 tablet_service.cc:1552] Processing DeleteTablet for tablet 033c2c016f8f4c01adc8bdbf2885e394 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f0bb5daf5c04698bad02fb5fa2e987e) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:03.316573 14576 tablet_service.cc:1505] Processing CreateTablet for tablet 52ae270f69724ef4b11f8831906add57 (DEFAULT_TABLE table=default.Table_2a5d1e8d612c4fc5954ea3efc00eb00b [id=ad68b61e866c40f28e5dbd143ccfcb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.317636 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ae270f69724ef4b11f8831906add57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.321156 16241 raft_consensus.cc:493] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:03.321610 16241 raft_consensus.cc:515] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.323364 16241 leader_election.cc:290] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:03.324359 16241 consensus_queue.cc:1048] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:03.324625 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "476eee1606e74832abf928dbaafc844d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:03.325449 14441 raft_consensus.cc:2468] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
W20251211 08:58:03.329608 13997 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d55c67cd21d141a593e3889032490dbc (table default.Table_781755712e1c4e5eb0414cdfd1117e54 [id=a762eb04798c4cffb074ba0e4559ead1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d55c67cd21d141a593e3889032490dbc consensus state has no leader
I20251211 08:58:03.329475 14231 leader_election.cc:304] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:58:03.330197 16241 raft_consensus.cc:2804] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:03.330519 16241 raft_consensus.cc:493] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:03.330868 16241 raft_consensus.cc:3060] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:03.330924 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b2445f66be54d3fbdc3be964b9a9132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:56 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:03.330614 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 with delete_type TABLET_DATA_DELETED (replaced by tablet e62ae5b540aa4680b4d18fee29c39e64) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:03.334007 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 6874fb018bc844b3aeeab7db8fc53e9a with delete_type TABLET_DATA_DELETED (replaced by tablet 593a90d88b194bb5a1c37e0e7e9097ef) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:03.338999 14425 tablet_service.cc:1552] Processing DeleteTablet for tablet 476eee1606e74832abf928dbaafc844d with delete_type TABLET_DATA_DELETED (replaced by tablet 62c8dcb36e694cf896bd1283db63b4cc) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:03.346287 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 616c9de4359c4589aa85ac4a7df4bf3f (DEFAULT_TABLE table=default.Table_ac7c79920e5d4e0fbb1298e4533e545f [id=2ff8d3aca2da4fde98599cdcb76f811f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.347266 14263 tablet_service.cc:1552] Processing DeleteTablet for tablet 080d1700df4a4df09acda65c6d021d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:03.348353 14425 tablet_service.cc:1505] Processing CreateTablet for tablet 616c9de4359c4589aa85ac4a7df4bf3f (DEFAULT_TABLE table=default.Table_ac7c79920e5d4e0fbb1298e4533e545f [id=2ff8d3aca2da4fde98599cdcb76f811f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.348738 14263 tablet_service.cc:1552] Processing DeleteTablet for tablet c341b762afd8440bb7198ca134c8dffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:37 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:03.349421 14425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 616c9de4359c4589aa85ac4a7df4bf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.350304 14414 tablet_service.cc:1505] Processing CreateTablet for tablet 52ae270f69724ef4b11f8831906add57 (DEFAULT_TABLE table=default.Table_2a5d1e8d612c4fc5954ea3efc00eb00b [id=ad68b61e866c40f28e5dbd143ccfcb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.352526 14263 tablet_service.cc:1552] Processing DeleteTablet for tablet dd14de369e084828afb8a924c46c649b with delete_type TABLET_DATA_DELETED (replaced by tablet ee50f29bfe674611a834d84a7c4e9070) from {username='slave'} at 127.0.0.1:46844
W20251211 08:58:03.329165 14012 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 63) took 3352 ms (client timeout 1000 ms). Trace:
I20251211 08:58:03.354959 14722 tablet_bootstrap.cc:700] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b: Time spent opening tablet: real 0.170s user 0.003s sys 0.000s
I20251211 08:58:03.355516 14722 tablet_bootstrap.cc:654] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:03.337661 16268 catalog_manager.cc:1269] Loaded cluster ID: ad83da70c3aa44c788d5a566bdc1630f
I20251211 08:58:03.366531 16268 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: loading cluster ID for follower catalog manager: success
I20251211 08:58:03.372164 14263 tablet_service.cc:1505] Processing CreateTablet for tablet ae62bc26539c41c8bc60497c08b5d5da (DEFAULT_TABLE table=default.Table_756c31646dff4edfba939b6fed9f48c3 [id=25b2ecb484e04eac9f58f7c49869130c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.372565 16241 raft_consensus.cc:515] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.388945 14279 tablet_service.cc:1505] Processing CreateTablet for tablet df205bd40e3941f590777dfcf5eb65d2 (DEFAULT_TABLE table=default.Table_415671e3e703478eb0cf348eb2943243 [id=547f65de592741bd96d87478e28c473f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.392920 14267 tablet_service.cc:1505] Processing CreateTablet for tablet c9975bfc37b14461be5e3066918dac0c (DEFAULT_TABLE table=default.Table_5df60df88ddd40deb8310ab9f413aab5 [id=baccb991ba0342118a78a2617bfb4c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.398211 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 616c9de4359c4589aa85ac4a7df4bf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.401223 14263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae62bc26539c41c8bc60497c08b5d5da. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:58:03.402881 14005 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 68) took 2423 ms (client timeout 1000 ms). Trace:
I20251211 08:58:03.404776 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df205bd40e3941f590777dfcf5eb65d2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:58:03.405818 14006 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 64) took 1806 ms (client timeout 1000 ms). Trace:
I20251211 08:58:03.352151 16285 raft_consensus.cc:493] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:03.407334 16285 raft_consensus.cc:515] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.409264 16285 leader_election.cc:290] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:03.371800 14413 tablet_service.cc:1505] Processing CreateTablet for tablet ae62bc26539c41c8bc60497c08b5d5da (DEFAULT_TABLE table=default.Table_756c31646dff4edfba939b6fed9f48c3 [id=25b2ecb484e04eac9f58f7c49869130c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.409889 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "680c8f54e75a4291bb4d769d94e21552" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:03.410169 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae62bc26539c41c8bc60497c08b5d5da. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:58:03.411088 14385 leader_election.cc:343] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:03.411916 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9975bfc37b14461be5e3066918dac0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.383544 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "476eee1606e74832abf928dbaafc844d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:03.415041 14449 raft_consensus.cc:3060] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:03.386698 14419 tablet_service.cc:1505] Processing CreateTablet for tablet df205bd40e3941f590777dfcf5eb65d2 (DEFAULT_TABLE table=default.Table_415671e3e703478eb0cf348eb2943243 [id=547f65de592741bd96d87478e28c473f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.394878 14423 tablet_service.cc:1505] Processing CreateTablet for tablet c9975bfc37b14461be5e3066918dac0c (DEFAULT_TABLE table=default.Table_5df60df88ddd40deb8310ab9f413aab5 [id=baccb991ba0342118a78a2617bfb4c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.416571 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9975bfc37b14461be5e3066918dac0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.419552 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df205bd40e3941f590777dfcf5eb65d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.424352 14607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5884dc6ef814552893cdf9dccfcce19" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:03.428251 16301 raft_consensus.cc:2804] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:03.429073 16301 raft_consensus.cc:697] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:03.432718 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet e9905227c0a844d199362a442e7d630c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:03.422714 14414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ae270f69724ef4b11f8831906add57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.443696 14714 tablet_bootstrap.cc:492] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:03.448513 16241 leader_election.cc:290] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:03.451472 14714 tablet_bootstrap.cc:654] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:03.453840 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet 570c7f08c46e4273bd38c32593dcf304 with delete_type TABLET_DATA_DELETED (replaced by tablet 3838fd220d034b3595f351b906188339) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:03.447088 16315 raft_consensus.cc:493] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:03.470252 16315 raft_consensus.cc:515] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:03.470964 14600 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s user 0.010s sys 0.013s
I20251211 08:58:03.472544 16315 leader_election.cc:290] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:58:03.472764 15758 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.61:34977, user_credentials={real_user=slave}} blocked reactor thread for 202287us
I20251211 08:58:03.474539 14579 tablet_service.cc:1505] Processing CreateTablet for tablet 6cecac6803124fb3a1ae254bd4b35225 (DEFAULT_TABLE table=default.Table_6856babaa0d64d45902ac1bf4c4f8b0e [id=caca4123579b4f91b7f8f2a046ecd999]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.476372 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cecac6803124fb3a1ae254bd4b35225. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.478706 14588 tablet_service.cc:1505] Processing CreateTablet for tablet baf86be1945043a6bf5e8e7f4ecb8ede (DEFAULT_TABLE table=default.Table_38faf33a40be4a6fa97f912efdc62b1b [id=53bb113fbaa04d469ef4bed9fffa9ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.479950 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baf86be1945043a6bf5e8e7f4ecb8ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.481453 14597 raft_consensus.cc:3060] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:03.481727 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825d8cf806d94702bdd11e0d63ee0c59" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:03.484432 16268 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: acquiring CA information for follower catalog manager: success
I20251211 08:58:03.488176 14582 tablet_service.cc:1505] Processing CreateTablet for tablet de6c58a36e5e4f15b41cbe8d14d30a22 (DEFAULT_TABLE table=default.Table_f7dd35033be14c598ae0f9801daa1c4b [id=3e5bd5e839aa4b3fb67fa798c2be3b95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.489161 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de6c58a36e5e4f15b41cbe8d14d30a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.489925 14449 raft_consensus.cc:2468] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:58:03.490432 14904 ts_tablet_manager.cc:1929] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:03.490741 14904 log.cc:1199] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/c55408f39a414e2caf1db48e6f29eca7
I20251211 08:58:03.491429 14904 ts_tablet_manager.cc:1950] T c55408f39a414e2caf1db48e6f29eca7 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:03.491482 14586 tablet_service.cc:1505] Processing CreateTablet for tablet a8b666397a5d4cac9fec1949f47e42c4 (DEFAULT_TABLE table=default.Table_07dddc710b174f5ca438600d9fe40b91 [id=c847a55379bd4fb99557df214b499910]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.499550 14577 tablet_service.cc:1505] Processing CreateTablet for tablet 4054b3b38a5144e1a6af165000a5946d (DEFAULT_TABLE table=default.Table_6229702ff48d405ab70d847e8418262a [id=124269cd63254dcf8ef373fb3f0d7f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.501137 14581 tablet_service.cc:1505] Processing CreateTablet for tablet 716295d94c884a3c9cae77200b611878 (DEFAULT_TABLE table=default.Table_cd8592260266488598378cde91871ad9 [id=ff13003e3a294a2e8acbae3bd9d645d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.502529 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 960e6bf66ad84af3a68ac9047c4d7f83 (DEFAULT_TABLE table=default.Table_0275161db5ae45b6950bd8b95723b20b [id=e1e1d62a8477419d9139c9606f23be0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.430094 16301 consensus_queue.cc:237] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:03.503825 14585 tablet_service.cc:1505] Processing CreateTablet for tablet 58d06c7007354844bdaaf3d26871b5ab (DEFAULT_TABLE table=default.Table_507f5535a8db4b5ba1c1d46b38a612ae [id=dc5b8bcdd735479eace6c7d56adc139b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.505751 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8b666397a5d4cac9fec1949f47e42c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.513828 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet f785a3cc2e8145eebae7e9c2885b0b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:03.525918 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4054b3b38a5144e1a6af165000a5946d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.526554 16322 tablet.cc:1722] T cb2ff90b54544420b1aad9d2ab4132ff P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:58:03.530303 14276 tablet_service.cc:1552] Processing DeleteTablet for tablet c112dfcc301648cda6f0f038b1f2af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:03.547076 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716295d94c884a3c9cae77200b611878. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.547821 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d06c7007354844bdaaf3d26871b5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.549614 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960e6bf66ad84af3a68ac9047c4d7f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:03.552697 14714 tablet_bootstrap.cc:492] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:03.553061 14714 ts_tablet_manager.cc:1403] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.110s user 0.010s sys 0.002s
I20251211 08:58:03.576696 15030 tablet_bootstrap.cc:492] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:03.579087 14574 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42846 (request call id 554) took 1003 ms. Trace:
I20251211 08:58:03.583220 14574 rpcz_store.cc:276] 1211 08:58:02.575392 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:02.575532 (+ 140us) service_pool.cc:225] Handling call
1211 08:58:02.576955 (+ 1423us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1211 08:58:02.577025 (+ 70us) ts_tablet_manager.cc:682] Creating new metadata...
1211 08:58:03.008841 (+431816us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:03.310326 (+301485us) tablet_replica.cc:206] Creating consensus instance
1211 08:58:03.578898 (+268572us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:58:03.584300 14714 raft_consensus.cc:359] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.584853 14714 raft_consensus.cc:385] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:03.585079 14714 raft_consensus.cc:740] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:03.585649 14714 consensus_queue.cc:260] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.588287 14714 ts_tablet_manager.cc:1434] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.035s user 0.006s sys 0.001s
I20251211 08:58:03.589272 14714 tablet_bootstrap.cc:492] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:03.577332 15030 ts_tablet_manager.cc:1403] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.470s user 0.015s sys 0.000s
I20251211 08:58:03.617286 15030 raft_consensus.cc:359] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:03.618147 15030 raft_consensus.cc:385] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:03.618474 15030 raft_consensus.cc:740] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:03.619513 14905 ts_tablet_manager.cc:1929] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251211 08:58:03.623698 14905 log.cc:1199] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/c55408f39a414e2caf1db48e6f29eca7
I20251211 08:58:03.624558 14905 ts_tablet_manager.cc:1950] T c55408f39a414e2caf1db48e6f29eca7 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:58:03.635506 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 323423us
W20251211 08:58:03.638875 14544 consensus_peers.cc:597] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:03.646422 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet ed0e90efe651499c929b1fb992712b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:03.647238 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet 51c3f5eaef784e868e565f81bd51a5e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:03.648026 14278 tablet_service.cc:1552] Processing DeleteTablet for tablet da57b72207414512b9744b283bc869fe with delete_type TABLET_DATA_DELETED (replaced by tablet 27cb4228204c4176b1a348960efd5b26) from {username='slave'} at 127.0.0.1:46844
W20251211 08:58:03.354285 14012 rpcz_store.cc:269] 1211 08:57:59.977347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.999845 (+ 22498us) service_pool.cc:225] Handling call
1211 08:58:02.515065 (+2515220us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:02.515107 (+ 42us) write_op.cc:454] Released schema lock
1211 08:58:03.329008 (+813901us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:02.497393 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:02.497727 (+ 334us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:02.497749 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:58:02.497771 (+ 22us) tablet.cc:660] Decoding operations
1211 08:58:02.497981 (+ 210us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:02.498023 (+ 42us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:02.498042 (+ 19us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:02.498178 (+ 136us) tablet.cc:699] Row locks acquired
1211 08:58:02.498191 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:58:02.498291 (+ 100us) write_op.cc:270] Start()
1211 08:58:02.498359 (+ 68us) write_op.cc:276] Timestamp: P: 1765443482498422 usec, L: 0
1211 08:58:02.498372 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:58:02.499216 (+ 844us) log.cc:844] Serialized 530 byte log entry
1211 08:58:02.507455 (+ 8239us) op_driver.cc:464] REPLICATION: finished
1211 08:58:02.507751 (+ 296us) write_op.cc:301] APPLY: starting
1211 08:58:02.507795 (+ 44us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:02.507870 (+ 75us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:02.507946 (+ 76us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:02.508117 (+ 171us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:02.508226 (+ 109us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:02.508269 (+ 43us) write_op.cc:312] APPLY: finished
1211 08:58:02.508407 (+ 138us) log.cc:844] Serialized 23 byte log entry
1211 08:58:02.508531 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:02.508589 (+ 58us) write_op.cc:454] Released schema lock
1211 08:58:02.508620 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1211 08:58:03.231016 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:03.231388 (+ 372us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:03.231417 (+ 29us) write_op.cc:435] Acquired schema lock
1211 08:58:03.231432 (+ 15us) tablet.cc:660] Decoding operations
1211 08:58:03.231678 (+ 246us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:03.231714 (+ 36us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:03.231740 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:03.231878 (+ 138us) tablet.cc:699] Row locks acquired
1211 08:58:03.231891 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:58:03.231991 (+ 100us) write_op.cc:270] Start()
1211 08:58:03.232066 (+ 75us) write_op.cc:276] Timestamp: P: 1765443483232029 usec, L: 0
1211 08:58:03.232081 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:58:03.232548 (+ 467us) log.cc:844] Serialized 356 byte log entry
1211 08:58:03.239839 (+ 7291us) op_driver.cc:464] REPLICATION: finished
1211 08:58:03.240158 (+ 319us) write_op.cc:301] APPLY: starting
1211 08:58:03.240235 (+ 77us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:03.240325 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:03.240418 (+ 93us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:03.240598 (+ 180us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:03.240736 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:03.240783 (+ 47us) write_op.cc:312] APPLY: finished
1211 08:58:03.240962 (+ 179us) log.cc:844] Serialized 23 byte log entry
1211 08:58:03.241128 (+ 166us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:03.241213 (+ 85us) write_op.cc:454] Released schema lock
1211 08:58:03.241243 (+ 30us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:03.241450 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:03.241475 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":183,"apply.run_cpu_time_us":1017,"apply.run_wall_time_us":1016,"num_ops":1,"prepare.queue_time_us":100,"prepare.run_cpu_time_us":2086,"prepare.run_wall_time_us":2084,"raft.queue_time_us":1167,"raft.run_cpu_time_us":556,"raft.run_wall_time_us":555,"replication_time_us":8987}],["op",{"apply.queue_time_us":185,"apply.run_cpu_time_us":1258,"apply.run_wall_time_us":1257,"num_ops":1,"prepare.queue_time_us":90,"prepare.run_cpu_time_us":1785,"prepare.run_wall_time_us":1783,"raft.queue_time_us":1062,"raft.run_cpu_time_us":596,"raft.run_wall_time_us":597,"replication_time_us":7668}]]}
I20251211 08:58:03.649004 16321 tablet.cc:1722] T cb2ff90b54544420b1aad9d2ab4132ff P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:58:03.650359 14012 master_service.cc:954] ReplaceTablet: received request to replace tablet c0e0ee99989d41c6af82e330ab8e353c from {username='slave'} at 127.0.0.1:55614
W20251211 08:58:03.651449 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 237837us
W20251211 08:58:03.406127 14006 rpcz_store.cc:269] 1211 08:58:01.599319 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:01.615655 (+ 16336us) service_pool.cc:225] Handling call
1211 08:58:03.405756 (+1790101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:02.923330 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:02.923654 (+ 324us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:02.923676 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:58:02.923688 (+ 12us) tablet.cc:660] Decoding operations
1211 08:58:02.923903 (+ 215us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:02.923936 (+ 33us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:02.923954 (+ 18us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:02.924063 (+ 109us) tablet.cc:699] Row locks acquired
1211 08:58:02.924085 (+ 22us) write_op.cc:260] PREPARE: finished
1211 08:58:02.924180 (+ 95us) write_op.cc:270] Start()
1211 08:58:02.924251 (+ 71us) write_op.cc:276] Timestamp: P: 1765443482924295 usec, L: 0
1211 08:58:02.924264 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:58:02.924700 (+ 436us) log.cc:844] Serialized 356 byte log entry
1211 08:58:02.930771 (+ 6071us) op_driver.cc:464] REPLICATION: finished
1211 08:58:02.931144 (+ 373us) write_op.cc:301] APPLY: starting
1211 08:58:02.931208 (+ 64us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:02.931306 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:02.931430 (+ 124us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:02.931611 (+ 181us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:02.931762 (+ 151us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:02.931817 (+ 55us) write_op.cc:312] APPLY: finished
1211 08:58:02.932004 (+ 187us) log.cc:844] Serialized 23 byte log entry
1211 08:58:02.932150 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:02.932221 (+ 71us) write_op.cc:454] Released schema lock
1211 08:58:02.932256 (+ 35us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:02.932507 (+ 251us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:02.932541 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":205,"apply.run_cpu_time_us":1326,"apply.run_wall_time_us":1325,"num_ops":1,"prepare.queue_time_us":10077,"prepare.run_cpu_time_us":1638,"prepare.run_wall_time_us":1638,"raft.queue_time_us":163,"raft.run_cpu_time_us":660,"raft.run_wall_time_us":659,"replication_time_us":6429}]]}
I20251211 08:58:03.654058 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_1610127d3bbc43ae803baa1e8470000f" } modify_external_catalogs: true
I20251211 08:58:03.654433 14231 leader_election.cc:304] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:58:03.654493 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_1610127d3bbc43ae803baa1e8470000f" } modify_external_catalogs: true
I20251211 08:58:03.656481 16247 raft_consensus.cc:2804] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:03.656912 16247 raft_consensus.cc:697] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
W20251211 08:58:03.403177 14005 rpcz_store.cc:269] 1211 08:58:00.979819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:01.019984 (+ 40165us) service_pool.cc:225] Handling call
1211 08:58:01.066658 (+ 46674us) spinlock_profiling.cc:243] Waited 39.4 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:58:03.402819 (+2336161us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:02.716648 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:02.717165 (+ 517us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:02.717197 (+ 32us) write_op.cc:435] Acquired schema lock
1211 08:58:02.717215 (+ 18us) tablet.cc:660] Decoding operations
1211 08:58:02.717511 (+ 296us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:02.717566 (+ 55us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:02.717594 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:02.717767 (+ 173us) tablet.cc:699] Row locks acquired
1211 08:58:02.717786 (+ 19us) write_op.cc:260] PREPARE: finished
1211 08:58:02.717912 (+ 126us) write_op.cc:270] Start()
1211 08:58:02.718010 (+ 98us) write_op.cc:276] Timestamp: P: 1765443482717935 usec, L: 0
1211 08:58:02.718032 (+ 22us) op_driver.cc:348] REPLICATION: starting
1211 08:58:02.718594 (+ 562us) log.cc:844] Serialized 530 byte log entry
1211 08:58:02.729044 (+ 10450us) op_driver.cc:464] REPLICATION: finished
1211 08:58:02.730467 (+ 1423us) write_op.cc:301] APPLY: starting
1211 08:58:02.730520 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:02.730618 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:02.730710 (+ 92us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:02.730906 (+ 196us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:02.731038 (+ 132us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:02.731099 (+ 61us) write_op.cc:312] APPLY: finished
1211 08:58:02.731282 (+ 183us) log.cc:844] Serialized 23 byte log entry
1211 08:58:02.731451 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:02.731527 (+ 76us) write_op.cc:454] Released schema lock
1211 08:58:02.731571 (+ 44us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:02.731800 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:02.731825 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":44415,"reported_tablets":24,"spinlock_wait_cycles":90586880,"child_traces":[["op",{"apply.queue_time_us":1266,"apply.run_cpu_time_us":1306,"apply.run_wall_time_us":1302,"num_ops":1,"prepare.queue_time_us":114,"prepare.run_cpu_time_us":2276,"prepare.run_wall_time_us":3292,"raft.queue_time_us":1220,"raft.run_cpu_time_us":629,"raft.run_wall_time_us":627,"replication_time_us":10923}]]}
I20251211 08:58:03.657629 16247 consensus_queue.cc:237] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.659117 14005 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_952b121ba0144ec3975634134ac3c07d" } modify_external_catalogs: true
I20251211 08:58:03.659615 14005 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_952b121ba0144ec3975634134ac3c07d" } modify_external_catalogs: true
W20251211 08:58:03.465269 14010 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 67) took 3837 ms (client timeout 1000 ms). Trace:
I20251211 08:58:03.532090 14013 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55614 (request call id 641) took 2661 ms. Trace:
I20251211 08:58:03.667199 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c02cb6ec01041dba118683a57aecde2" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:03.665271 14013 rpcz_store.cc:276] 1211 08:58:00.870902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.871023 (+ 121us) service_pool.cc:225] Handling call
1211 08:58:00.903077 (+ 32054us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da87016 00007fc20b6b5b24 00007fc20b6b5ab2 00007fc20b6b59e1 00007fc20b6b595c 00007fc20b6b4c12 00007fc20998455c 00007fc209983996
1211 08:58:00.903713 (+ 636us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:58:00.904279 (+ 566us) catalog_manager.cc:3708] Apply alter schema
1211 08:58:00.904739 (+ 460us) catalog_manager.cc:3740] Acquired catalog manager lock
1211 08:58:00.904907 (+ 168us) catalog_manager.cc:3898] Updating metadata on disk
1211 08:58:01.173752 (+268845us) catalog_manager.cc:3929] Committing alterations to in-memory state
1211 08:58:03.532024 (+2358272us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:01.037790 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:01.038152 (+ 362us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:01.038174 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:58:01.038185 (+ 11us) tablet.cc:660] Decoding operations
1211 08:58:01.038444 (+ 259us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:01.038483 (+ 39us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:01.038506 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:01.038653 (+ 147us) tablet.cc:699] Row locks acquired
1211 08:58:01.038666 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:58:01.038784 (+ 118us) write_op.cc:270] Start()
1211 08:58:01.038856 (+ 72us) write_op.cc:276] Timestamp: P: 1765443481038728 usec, L: 0
1211 08:58:01.038871 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:58:01.131506 (+ 92635us) log.cc:844] Serialized 405 byte log entry
1211 08:58:01.152003 (+ 20497us) spinlock_profiling.cc:243] Waited 92.2 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbef5a1 00007fc20bc81aa8 00007fc20c0f34d1 00007fc20c0f2ca8 00007fc20c0f7941 00007fc20c0f78f9 00007fc20c0f7889 00007fc20c0f7851 00007fc20c0f6a7d 00000000005aa2b4 00000000005aa0e9
1211 08:58:01.160774 (+ 8771us) op_driver.cc:464] REPLICATION: finished
1211 08:58:01.170583 (+ 9809us) write_op.cc:301] APPLY: starting
1211 08:58:01.170637 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:01.170728 (+ 91us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:01.170748 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:01.171083 (+ 335us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:01.171229 (+ 146us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:01.171280 (+ 51us) write_op.cc:312] APPLY: finished
1211 08:58:01.171487 (+ 207us) log.cc:844] Serialized 23 byte log entry
1211 08:58:01.171654 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:01.171708 (+ 54us) write_op.cc:454] Released schema lock
1211 08:58:01.171754 (+ 46us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:01.171966 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:01.171990 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":69556992,"child_traces":[["op",{"apply.queue_time_us":9644,"apply.run_cpu_time_us":1260,"apply.run_wall_time_us":1340,"num_ops":1,"prepare.queue_time_us":123773,"prepare.run_cpu_time_us":3333,"prepare.run_wall_time_us":114405,"raft.queue_time_us":81,"raft.run_cpu_time_us":570,"raft.run_wall_time_us":569,"replication_time_us":121825,"spinlock_wait_cycles":211980544}]]}
I20251211 08:58:03.669538 14013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_07f6651621854d2f848120541ef54d4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:03.672363 14722 tablet_bootstrap.cc:492] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:03.672750 14722 ts_tablet_manager.cc:1403] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.542s user 0.009s sys 0.000s
I20251211 08:58:03.674218 16268 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251211 08:58:03.680938 14714 tablet_bootstrap.cc:654] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:58:03.684666 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 46012.6us
I20251211 08:58:03.623150 15030 consensus_queue.cc:260] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:58:03.693789 14382 leader_election.cc:343] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:03.694130 14382 leader_election.cc:304] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:03.695466 16306 raft_consensus.cc:2749] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:03.696254 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 499) took 4850 ms. Trace:
I20251211 08:58:03.696563 14904 rpcz_store.cc:276] 1211 08:57:58.845624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.845747 (+ 123us) service_pool.cc:225] Handling call
1211 08:58:02.937660 (+4091913us) ts_tablet_manager.cc:1151] Deleting tablet c55408f39a414e2caf1db48e6f29eca7
1211 08:58:03.109162 (+171502us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.261095 (+151933us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:03.490225 (+229130us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:03.696198 (+205973us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4076121}
I20251211 08:58:03.697953 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 500) took 4825 ms. Trace:
I20251211 08:58:03.698230 14904 rpcz_store.cc:276] 1211 08:57:58.872449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.872560 (+ 111us) service_pool.cc:225] Handling call
1211 08:58:03.697596 (+4825036us) ts_tablet_manager.cc:1151] Deleting tablet 2bf678f74a12464f9361e379e8af0522
1211 08:58:03.697640 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.697889 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4824558}
W20251211 08:58:03.662528 14010 rpcz_store.cc:269] 1211 08:57:59.628353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.711506 (+ 83153us) service_pool.cc:225] Handling call
1211 08:57:59.762051 (+ 50545us) spinlock_profiling.cc:243] Waited 41.9 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:58:03.465209 (+3703158us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:02.181440 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:02.181887 (+ 447us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:02.181907 (+ 20us) write_op.cc:435] Acquired schema lock
1211 08:58:02.181920 (+ 13us) tablet.cc:660] Decoding operations
1211 08:58:02.182134 (+ 214us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:02.182169 (+ 35us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:02.182189 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:02.182318 (+ 129us) tablet.cc:699] Row locks acquired
1211 08:58:02.182332 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:58:02.182413 (+ 81us) write_op.cc:270] Start()
1211 08:58:02.182484 (+ 71us) write_op.cc:276] Timestamp: P: 1765443482182427 usec, L: 0
1211 08:58:02.182499 (+ 15us) op_driver.cc:348] REPLICATION: starting
1211 08:58:02.183011 (+ 512us) log.cc:844] Serialized 530 byte log entry
1211 08:58:02.193151 (+ 10140us) op_driver.cc:464] REPLICATION: finished
1211 08:58:02.240475 (+ 47324us) write_op.cc:301] APPLY: starting
1211 08:58:02.240599 (+ 124us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:02.240852 (+ 253us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:02.240950 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:02.241173 (+ 223us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:02.241300 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:02.241341 (+ 41us) write_op.cc:312] APPLY: finished
1211 08:58:02.241507 (+ 166us) log.cc:844] Serialized 23 byte log entry
1211 08:58:02.241654 (+ 147us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:02.241709 (+ 55us) write_op.cc:454] Released schema lock
1211 08:58:02.241737 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":246673,"reported_tablets":29,"spinlock_wait_cycles":96357632,"child_traces":[["op",{"apply.queue_time_us":47082,"apply.run_cpu_time_us":1505,"apply.run_wall_time_us":1505,"num_ops":1,"prepare.queue_time_us":89,"prepare.run_cpu_time_us":1768,"prepare.run_wall_time_us":2636,"replication_time_us":10567,"thread_start_us":80314,"threads_started":1}]]}
I20251211 08:58:03.700837 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet c55408f39a414e2caf1db48e6f29eca7 (table default.Table_2a5d1e8d612c4fc5954ea3efc00eb00b [id=ad68b61e866c40f28e5dbd143ccfcb55]) successfully deleted
I20251211 08:58:03.700661 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_7ff5b865284f49c6826753de45156314"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:03.702001 14010 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ff5b865284f49c6826753de45156314 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:03.701931 14722 raft_consensus.cc:359] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:03.702510 14722 raft_consensus.cc:385] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:03.702718 14722 raft_consensus.cc:740] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
W20251211 08:58:03.703837 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 2bf678f74a12464f9361e379e8af0522 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bf678f74a12464f9361e379e8af0522 already in progress: creating tablet
I20251211 08:58:03.710218 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 511) took 4320 ms. Trace:
W20251211 08:58:03.645803 14011 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 98) took 3426 ms (client timeout 1000 ms). Trace:
W20251211 08:58:03.711318 14011 rpcz_store.cc:269] 1211 08:58:00.219677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.222896 (+ 3219us) service_pool.cc:225] Handling call
1211 08:58:00.324080 (+101184us) spinlock_profiling.cc:243] Waited 77.7 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:58:03.645742 (+3321662us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":29,"spinlock_wait_cycles":178649856}
I20251211 08:58:03.712392 14011 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_52f6bbd3c66a47e7949baa7892ea2e67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:03.713881 14011 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52f6bbd3c66a47e7949baa7892ea2e67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:03.715090 14904 rpcz_store.cc:276] 1211 08:57:59.389553 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.389656 (+ 103us) service_pool.cc:225] Handling call
1211 08:58:03.700514 (+4310858us) ts_tablet_manager.cc:1151] Deleting tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8
1211 08:58:03.704709 (+ 4195us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.710154 (+ 5445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4310387}
W20251211 08:58:03.717475 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 already in progress: creating tablet
W20251211 08:58:03.671238 14013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07f6651621854d2f848120541ef54d4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:03.712507 14722 consensus_queue.cc:260] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:03.735879 14722 ts_tablet_manager.cc:1434] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.063s user 0.005s sys 0.000s
I20251211 08:58:03.744805 16320 ts_tablet_manager.cc:1929] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:03.770183 16320 log.cc:1199] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/033c2c016f8f4c01adc8bdbf2885e394
I20251211 08:58:03.771950 16320 ts_tablet_manager.cc:1950] T 033c2c016f8f4c01adc8bdbf2885e394 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:03.760354 14714 tablet_bootstrap.cc:492] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:03.774188 14714 ts_tablet_manager.cc:1403] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.185s user 0.011s sys 0.000s
I20251211 08:58:03.776664 14714 raft_consensus.cc:359] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.779589 14714 raft_consensus.cc:385] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:03.779644 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 033c2c016f8f4c01adc8bdbf2885e394 (table default.Table_1ce47a525fa9427198a4b8c589d662c7 [id=c7c35cc13c2840fe969a649388e5d250]) successfully deleted
I20251211 08:58:03.779912 14714 raft_consensus.cc:740] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:03.786998 14714 consensus_queue.cc:260] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.789108 14714 ts_tablet_manager.cc:1434] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.015s user 0.000s sys 0.004s
I20251211 08:58:03.795782 16320 tablet_replica.cc:333] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:03.796605 16320 raft_consensus.cc:2243] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:03.797458 16320 raft_consensus.cc:2272] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:03.798627 14714 tablet_bootstrap.cc:492] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:03.806035 16320 ts_tablet_manager.cc:1916] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:03.821566 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 512) took 4431 ms. Trace:
I20251211 08:58:03.821874 14904 rpcz_store.cc:276] 1211 08:57:59.389839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.389923 (+ 84us) service_pool.cc:225] Handling call
1211 08:58:03.715906 (+4325983us) ts_tablet_manager.cc:1151] Deleting tablet f6fb2096f80a43b5a5a6230a824fceb7
1211 08:58:03.821264 (+105358us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.821511 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4325554}
I20251211 08:58:03.822899 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 518) took 3601 ms. Trace:
I20251211 08:58:03.823149 14904 rpcz_store.cc:276] 1211 08:58:00.221179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.227770 (+ 6591us) service_pool.cc:225] Handling call
1211 08:58:03.822596 (+3594826us) ts_tablet_manager.cc:1151] Deleting tablet ddbc5b3fad2e43b481ff733b5bad189a
1211 08:58:03.822637 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.822853 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3594081}
I20251211 08:58:03.823969 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 520) took 3352 ms. Trace:
I20251211 08:58:03.824208 14904 rpcz_store.cc:276] 1211 08:58:00.471956 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.472067 (+ 111us) service_pool.cc:225] Handling call
1211 08:58:03.823768 (+3351701us) ts_tablet_manager.cc:1151] Deleting tablet c6d0fef50524476dbd47fc70e4bda319
1211 08:58:03.823797 (+ 29us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.823933 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3351065}
I20251211 08:58:03.826561 14722 tablet_bootstrap.cc:492] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:58:03.827567 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet f6fb2096f80a43b5a5a6230a824fceb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6fb2096f80a43b5a5a6230a824fceb7 already in progress: creating tablet
W20251211 08:58:03.828184 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet ddbc5b3fad2e43b481ff733b5bad189a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddbc5b3fad2e43b481ff733b5bad189a already in progress: creating tablet
W20251211 08:58:03.828676 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet c6d0fef50524476dbd47fc70e4bda319 because the tablet was not found. No further retry: Not found: Tablet not found: c6d0fef50524476dbd47fc70e4bda319
I20251211 08:58:03.829378 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c02cb6ec01041dba118683a57aecde2" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:03.830180 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 521) took 3345 ms. Trace:
I20251211 08:58:03.830487 14904 rpcz_store.cc:276] 1211 08:58:00.484363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.484463 (+ 100us) service_pool.cc:225] Handling call
1211 08:58:03.824876 (+3340413us) ts_tablet_manager.cc:1151] Deleting tablet 8e05628011634149b17d298c54074d00
1211 08:58:03.824910 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.830127 (+ 5217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3339657}
I20251211 08:58:03.832000 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 522) took 3312 ms. Trace:
I20251211 08:58:03.832720 14904 rpcz_store.cc:276] 1211 08:58:00.519765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.519953 (+ 188us) service_pool.cc:225] Handling call
1211 08:58:03.831799 (+3311846us) ts_tablet_manager.cc:1151] Deleting tablet 07aef1eea7c04d9cbf35c82e45df7a16
1211 08:58:03.831833 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.831963 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3294148}
W20251211 08:58:03.833793 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 8e05628011634149b17d298c54074d00 because the tablet was not found. No further retry: Not found: Tablet not found: 8e05628011634149b17d298c54074d00
W20251211 08:58:03.833935 14009 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 69) took 1710 ms (client timeout 1000 ms). Trace:
W20251211 08:58:03.834299 14009 rpcz_store.cc:269] 1211 08:58:02.123461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:02.231275 (+107814us) service_pool.cc:225] Handling call
1211 08:58:02.278646 (+ 47371us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:58:03.833857 (+1555211us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":117612,"reported_tablets":28,"spinlock_wait_cycles":48723200}
I20251211 08:58:03.844254 14904 tablet_replica.cc:333] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:03.845188 14904 raft_consensus.cc:2243] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251211 08:58:03.845958 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 07aef1eea7c04d9cbf35c82e45df7a16 because the tablet was not found. No further retry: Not found: Tablet not found: 07aef1eea7c04d9cbf35c82e45df7a16
I20251211 08:58:03.851009 14904 raft_consensus.cc:2272] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:03.858167 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet f2f095b59ecf44ffabaa0ac22212938b with delete_type TABLET_DATA_DELETED (replaced by tablet b3ad595efe5241e8a4f40b16be2ccd63) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:03.864719 14904 ts_tablet_manager.cc:1916] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:03.872539 16340 raft_consensus.cc:493] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:03.872946 16340 raft_consensus.cc:515] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.873699 14722 tablet_bootstrap.cc:654] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:03.874686 16340 leader_election.cc:290] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:58:03.876309 14231 leader_election.cc:343] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 570c7f08c46e4273bd38c32593dcf304
I20251211 08:58:03.880185 15030 ts_tablet_manager.cc:1434] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.265s user 0.004s sys 0.001s
I20251211 08:58:03.881129 15030 tablet_bootstrap.cc:492] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:03.900115 14714 tablet_bootstrap.cc:654] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:58:03.921772 14382 leader_election.cc:343] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:03.922226 14382 leader_election.cc:304] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:03.923183 16307 raft_consensus.cc:2749] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:03.926191 14722 tablet_bootstrap.cc:492] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:03.926640 14722 ts_tablet_manager.cc:1403] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.100s user 0.012s sys 0.005s
I20251211 08:58:03.928900 14722 raft_consensus.cc:359] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:03.929492 14722 raft_consensus.cc:385] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:03.929764 14722 raft_consensus.cc:740] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:03.936383 16340 consensus_queue.cc:1048] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:03.943011 14722 consensus_queue.cc:260] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:03.946321 16326 consensus_queue.cc:1048] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251211 08:58:03.948220 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 511) took 5103 ms. Trace:
I20251211 08:58:03.948613 14714 tablet_bootstrap.cc:492] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:03.948510 14905 rpcz_store.cc:276] 1211 08:57:58.844808 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.844918 (+ 110us) service_pool.cc:225] Handling call
1211 08:58:03.092565 (+4247647us) ts_tablet_manager.cc:1151] Deleting tablet c55408f39a414e2caf1db48e6f29eca7
1211 08:58:03.092603 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.128431 (+ 35828us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:03.619220 (+490789us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:03.948162 (+328942us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4161224}
I20251211 08:58:03.949035 14714 ts_tablet_manager.cc:1403] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.154s user 0.005s sys 0.004s
I20251211 08:58:03.955574 14714 raft_consensus.cc:359] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.956164 14714 raft_consensus.cc:385] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:03.956408 14714 raft_consensus.cc:740] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:03.957136 14722 ts_tablet_manager.cc:1434] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.030s user 0.004s sys 0.001s
I20251211 08:58:03.956949 14714 consensus_queue.cc:260] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:03.968168 15030 tablet_bootstrap.cc:654] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:03.982085 14722 tablet_bootstrap.cc:492] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:03.958750 14714 ts_tablet_manager.cc:1434] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20251211 08:58:03.978943 16307 raft_consensus.cc:493] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:03.993748 16307 raft_consensus.cc:515] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:03.994208 14714 tablet_bootstrap.cc:492] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:03.998211 14292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d8d84e594ea47f3bc9127cc587e8aa7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:03.998785 14292 raft_consensus.cc:2468] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:58:03.999716 14385 leader_election.cc:304] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:58:03.974689 14904 ts_tablet_manager.cc:1929] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:04.005919 14904 log.cc:1199] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/2a1b011cb7c647f79bb16ce1c92bcff5
I20251211 08:58:04.006565 14904 ts_tablet_manager.cc:1950] T 2a1b011cb7c647f79bb16ce1c92bcff5 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:04.011336 16307 leader_election.cc:290] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:04.014016 16306 raft_consensus.cc:2804] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:04.014350 16306 raft_consensus.cc:493] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:04.014689 16306 raft_consensus.cc:3060] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:04.016925 14714 tablet_bootstrap.cc:654] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:58:04.021521 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 46506.3us
W20251211 08:58:04.022233 14542 consensus_peers.cc:597] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 033c2c016f8f4c01adc8bdbf2885e394. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:04.027765 14722 tablet_bootstrap.cc:654] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:04.031239 16306 raft_consensus.cc:515] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:04.031303 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 519) took 3565 ms. Trace:
I20251211 08:58:04.033066 14904 rpcz_store.cc:276] 1211 08:58:00.465501 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.470159 (+ 4658us) service_pool.cc:225] Handling call
1211 08:58:03.844117 (+3373958us) ts_tablet_manager.cc:1151] Deleting tablet 2a1b011cb7c647f79bb16ce1c92bcff5
1211 08:58:03.844150 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:03.936438 (+ 92288us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:03.974457 (+ 38019us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.031251 (+ 56794us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3295438}
I20251211 08:58:04.034003 14904 tablet_replica.cc:333] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:04.034193 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 2a1b011cb7c647f79bb16ce1c92bcff5 (table default.Table_03bee763242749cfa5fb0fe7d7e7fb7f [id=209008ecbdd4441989221f0f956a87b6]) successfully deleted
I20251211 08:58:04.042739 16312 consensus_queue.cc:1048] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20251211 08:58:04.044770 14541 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.000s sys 0.011s
W20251211 08:58:04.045091 14541 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s user 0.000s sys 0.011s
I20251211 08:58:04.048202 14579 tablet_service.cc:1505] Processing CreateTablet for tablet df06100dbe4e43fc9840a8cc959d47bf (DEFAULT_TABLE table=default.Table_cc7fd4312d5d4dc7807e350123a637e0 [id=ae4b8a68a7f2484fa0980dae0d574de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:04.053609 14904 raft_consensus.cc:2243] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:04.054057 14904 raft_consensus.cc:2272] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:04.055794 14904 ts_tablet_manager.cc:1916] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:04.060595 14597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "680c8f54e75a4291bb4d769d94e21552" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:04.076475 14588 tablet_service.cc:1552] Processing DeleteTablet for tablet da52cace2aab4f9dacef18ec5fa56aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:51 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.102522 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.115352 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 512) took 5240 ms. Trace:
I20251211 08:58:04.115684 14905 rpcz_store.cc:276] 1211 08:57:58.874811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:58.875043 (+ 232us) service_pool.cc:225] Handling call
1211 08:58:03.949476 (+5074433us) ts_tablet_manager.cc:1151] Deleting tablet 2bf678f74a12464f9361e379e8af0522
1211 08:58:04.114985 (+165509us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.115301 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5014682}
I20251211 08:58:04.117189 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 523) took 4827 ms. Trace:
I20251211 08:58:04.117429 14905 rpcz_store.cc:276] 1211 08:57:59.289541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.289810 (+ 269us) service_pool.cc:225] Handling call
1211 08:58:04.116917 (+4827107us) ts_tablet_manager.cc:1151] Deleting tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8
1211 08:58:04.116953 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.117151 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4826592}
I20251211 08:58:04.118336 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 525) took 4779 ms. Trace:
I20251211 08:58:04.118582 14905 rpcz_store.cc:276] 1211 08:57:59.339132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:57:59.339259 (+ 127us) service_pool.cc:225] Handling call
1211 08:58:04.118057 (+4778798us) ts_tablet_manager.cc:1151] Deleting tablet f6fb2096f80a43b5a5a6230a824fceb7
1211 08:58:04.118092 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.118299 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4778375}
I20251211 08:58:04.120201 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 533) took 3874 ms. Trace:
I20251211 08:58:04.120483 14905 rpcz_store.cc:276] 1211 08:58:00.245543 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.246854 (+ 1311us) service_pool.cc:225] Handling call
1211 08:58:04.119847 (+3872993us) ts_tablet_manager.cc:1151] Deleting tablet ddbc5b3fad2e43b481ff733b5bad189a
1211 08:58:04.119896 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.120158 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3872517}
I20251211 08:58:04.121403 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 535) took 3843 ms. Trace:
I20251211 08:58:04.121557 14904 ts_tablet_manager.cc:1929] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:04.121856 14904 log.cc:1199] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/00e5c4bf71b44abca66b0e94995c533e
I20251211 08:58:04.121704 14905 rpcz_store.cc:276] 1211 08:58:00.277610 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.277757 (+ 147us) service_pool.cc:225] Handling call
1211 08:58:04.121162 (+3843405us) ts_tablet_manager.cc:1151] Deleting tablet e9905227c0a844d199362a442e7d630c
1211 08:58:04.121199 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.121355 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3842881}
I20251211 08:58:04.122385 14904 ts_tablet_manager.cc:1950] T 00e5c4bf71b44abca66b0e94995c533e P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:04.122799 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 536) took 3843 ms. Trace:
I20251211 08:58:04.123090 14905 rpcz_store.cc:276] 1211 08:58:00.279129 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.279240 (+ 111us) service_pool.cc:225] Handling call
1211 08:58:04.122505 (+3843265us) ts_tablet_manager.cc:1151] Deleting tablet f2f095b59ecf44ffabaa0ac22212938b
1211 08:58:04.122583 (+ 78us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.122728 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3842764}
I20251211 08:58:04.125931 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 537) took 3845 ms. Trace:
I20251211 08:58:04.126226 14905 rpcz_store.cc:276] 1211 08:58:00.280334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.280472 (+ 138us) service_pool.cc:225] Handling call
1211 08:58:04.125739 (+3845267us) ts_tablet_manager.cc:1151] Deleting tablet da57b72207414512b9744b283bc869fe
1211 08:58:04.125775 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.125897 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3844830}
I20251211 08:58:04.124409 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df06100dbe4e43fc9840a8cc959d47bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:04.135355 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 523) took 3405 ms. Trace:
I20251211 08:58:04.135722 14904 rpcz_store.cc:276] 1211 08:58:00.729882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.730001 (+ 119us) service_pool.cc:225] Handling call
1211 08:58:04.033877 (+3303876us) ts_tablet_manager.cc:1151] Deleting tablet 00e5c4bf71b44abca66b0e94995c533e
1211 08:58:04.033911 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.103876 (+ 69965us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.121360 (+ 17484us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.135309 (+ 13949us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18839,"tablet-delete.queue_time_us":3303261}
I20251211 08:58:04.136919 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 00e5c4bf71b44abca66b0e94995c533e (table default.Table_dcb6d015c5c543689fc117f18bdff260 [id=ff5165bdae864e59a46b71d1ca8d8611]) successfully deleted
I20251211 08:58:04.137682 14904 tablet_replica.cc:333] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:04.143362 14904 raft_consensus.cc:2243] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:04.144110 14904 raft_consensus.cc:2272] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:04.148578 14904 ts_tablet_manager.cc:1916] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:04.061195 16307 raft_consensus.cc:493] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:04.156160 16307 raft_consensus.cc:515] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.157858 16307 leader_election.cc:290] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:04.092353 14714 tablet_bootstrap.cc:492] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:04.158457 14714 ts_tablet_manager.cc:1403] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.164s user 0.010s sys 0.000s
I20251211 08:58:04.109896 16361 raft_consensus.cc:493] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:04.160207 16361 raft_consensus.cc:515] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.162266 16361 leader_election.cc:290] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:04.164031 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7264e13dd44b499683e32b3fc62ed8ed" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:04.166378 14714 raft_consensus.cc:359] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:04.167002 14714 raft_consensus.cc:385] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:04.167205 14714 raft_consensus.cc:740] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:04.167860 14714 consensus_queue.cc:260] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:03.873025 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
W20251211 08:58:04.168951 14385 leader_election.cc:343] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.169473 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:04.170308 14007 catalog_manager.cc:5654] T 0fdc020f21aa4f35880a435d7cfb605d P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } }
I20251211 08:58:04.171820 14714 ts_tablet_manager.cc:1434] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20251211 08:58:04.172166 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet d7a8acb51a2948f4a018787bfaecb694 (table default.Table_9e007a9469d54d99b9471cb8e653b620 [id=de3a242b7e6d480ebf509e458ec002c7]). Expected version 1 got 0
I20251211 08:58:04.172662 14714 tablet_bootstrap.cc:492] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:04.173182 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (table default.Table_ae8d82f4355240188c054f001b2c05a4 [id=f74e074787144a21ad31d4d89cc509c9]). Expected version 2 got 1
I20251211 08:58:04.173729 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:04.166385 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d8d84e594ea47f3bc9127cc587e8aa7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:04.179365 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 538) took 3897 ms. Trace:
I20251211 08:58:04.179458 14598 raft_consensus.cc:2468] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:58:04.179756 14905 rpcz_store.cc:276] 1211 08:58:00.281643 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.281760 (+ 117us) service_pool.cc:225] Handling call
1211 08:58:04.179042 (+3897282us) ts_tablet_manager.cc:1151] Deleting tablet 8e05628011634149b17d298c54074d00
1211 08:58:04.179113 (+ 71us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.179316 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3896852}
I20251211 08:58:04.143627 14581 tablet_service.cc:1552] Processing DeleteTablet for tablet 07aef1eea7c04d9cbf35c82e45df7a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:41 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.180945 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 540) took 3896 ms. Trace:
I20251211 08:58:04.138971 14588 tablet_service.cc:1505] Processing CreateTablet for tablet ae62bc26539c41c8bc60497c08b5d5da (DEFAULT_TABLE table=default.Table_756c31646dff4edfba939b6fed9f48c3 [id=25b2ecb484e04eac9f58f7c49869130c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:04.181203 14905 rpcz_store.cc:276] 1211 08:58:00.284298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.284536 (+ 238us) service_pool.cc:225] Handling call
1211 08:58:04.180668 (+3896132us) ts_tablet_manager.cc:1151] Deleting tablet e15483f539f24f1ba1b442663d66fd45
1211 08:58:04.180729 (+ 61us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.180902 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3895628}
I20251211 08:58:04.182053 14905 tablet_replica.cc:333] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:04.166767 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "947e6203f910442ca01e2c21587cfe90" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:04.184577 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 00e5c4bf71b44abca66b0e94995c533e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:45 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:04.158564 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet f3403116517f45c59f87fa5bda0b4071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.144313 16301 consensus_queue.cc:1048] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.042s
W20251211 08:58:04.185863 14008 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 99) took 2659 ms (client timeout 1000 ms). Trace:
W20251211 08:58:04.186153 14008 rpcz_store.cc:269] 1211 08:58:01.526588 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:01.532354 (+ 5766us) service_pool.cc:225] Handling call
1211 08:58:04.185809 (+2653455us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11754,"reported_tablets":31}
I20251211 08:58:04.135226 14577 tablet_service.cc:1552] Processing DeleteTablet for tablet c112dfcc301648cda6f0f038b1f2af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.138356 14573 tablet_service.cc:1505] Processing CreateTablet for tablet 616c9de4359c4589aa85ac4a7df4bf3f (DEFAULT_TABLE table=default.Table_ac7c79920e5d4e0fbb1298e4533e545f [id=2ff8d3aca2da4fde98599cdcb76f811f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:04.190357 14905 raft_consensus.cc:2243] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:04.190786 16361 raft_consensus.cc:493] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:04.190873 14573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 616c9de4359c4589aa85ac4a7df4bf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:04.190922 14905 raft_consensus.cc:2272] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:04.191165 16361 raft_consensus.cc:515] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.192922 16361 leader_election.cc:290] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:04.193015 14905 ts_tablet_manager.cc:1916] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:04.160049 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 570c7f08c46e4273bd38c32593dcf304 with delete_type TABLET_DATA_DELETED (replaced by tablet 3838fd220d034b3595f351b906188339) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.194804 14714 tablet_bootstrap.cc:654] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:04.197362 16306 leader_election.cc:290] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:04.197973 16361 raft_consensus.cc:493] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:04.198379 16361 raft_consensus.cc:515] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:04.209770 16361 leader_election.cc:290] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:04.209949 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae62bc26539c41c8bc60497c08b5d5da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:04.216432 14586 tablet_service.cc:1505] Processing CreateTablet for tablet df205bd40e3941f590777dfcf5eb65d2 (DEFAULT_TABLE table=default.Table_415671e3e703478eb0cf348eb2943243 [id=547f65de592741bd96d87478e28c473f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:04.217500 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df205bd40e3941f590777dfcf5eb65d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:04.225538 14582 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b2445f66be54d3fbdc3be964b9a9132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:56 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.226163 14574 tablet_service.cc:1552] Processing DeleteTablet for tablet 6874fb018bc844b3aeeab7db8fc53e9a with delete_type TABLET_DATA_DELETED (replaced by tablet 593a90d88b194bb5a1c37e0e7e9097ef) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.227380 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet f785a3cc2e8145eebae7e9c2885b0b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:38 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.228408 14581 tablet_service.cc:1552] Processing DeleteTablet for tablet c341b762afd8440bb7198ca134c8dffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:37 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.228967 14577 tablet_service.cc:1552] Processing DeleteTablet for tablet 45c7d841bd3643a799a8b96e82d6d675 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:49 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.230019 14580 tablet_service.cc:1505] Processing CreateTablet for tablet c9975bfc37b14461be5e3066918dac0c (DEFAULT_TABLE table=default.Table_5df60df88ddd40deb8310ab9f413aab5 [id=baccb991ba0342118a78a2617bfb4c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:03.993213 16320 ts_tablet_manager.cc:1929] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:04.231065 16320 log.cc:1199] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/ed0e90efe651499c929b1fb992712b96
I20251211 08:58:04.231936 16320 ts_tablet_manager.cc:1950] T ed0e90efe651499c929b1fb992712b96 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:04.234748 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet ed0e90efe651499c929b1fb992712b96 (table default.Table_eb3b1d93a3ec4891a9d33b131661fd65 [id=46237116be8e46a68f51e20e008f374b]) successfully deleted
I20251211 08:58:04.236085 14714 tablet_bootstrap.cc:492] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:04.236428 14714 ts_tablet_manager.cc:1403] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.001s
I20251211 08:58:04.238361 14714 raft_consensus.cc:359] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:04.238943 14714 raft_consensus.cc:385] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:04.239173 14714 raft_consensus.cc:740] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:04.242470 16355 tablet.cc:1722] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:58:04.250789 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9975bfc37b14461be5e3066918dac0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:04.251235 14714 consensus_queue.cc:260] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:04.258688 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7264e13dd44b499683e32b3fc62ed8ed" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:04.259545 14597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1065085c701f48c9ad7206c410ce5985" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:04.259675 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d8d84e594ea47f3bc9127cc587e8aa7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:58:04.260108 14604 raft_consensus.cc:3060] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:04.260780 14597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "476eee1606e74832abf928dbaafc844d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:04.261245 16369 tablet.cc:1722] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:58:04.262913 14597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "476eee1606e74832abf928dbaafc844d" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:58:04.265522 14598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a74d250ee85a4c768c76a8400003ccdb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:04.267622 14604 raft_consensus.cc:2468] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
I20251211 08:58:04.271371 15030 tablet_bootstrap.cc:492] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:04.271817 15030 ts_tablet_manager.cc:1403] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.391s user 0.004s sys 0.007s
I20251211 08:58:04.274808 14905 ts_tablet_manager.cc:1929] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:04.276288 14905 log.cc:1199] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/51c3f5eaef784e868e565f81bd51a5e4
I20251211 08:58:04.277431 14905 ts_tablet_manager.cc:1950] T 51c3f5eaef784e868e565f81bd51a5e4 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:04.295009 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a74d250ee85a4c768c76a8400003ccdb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:04.296002 14297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1065085c701f48c9ad7206c410ce5985" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:04.296826 14904 ts_tablet_manager.cc:1929] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:04.297194 14904 log.cc:1199] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/f3403116517f45c59f87fa5bda0b4071
W20251211 08:58:04.297857 14385 leader_election.cc:343] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.297992 14904 ts_tablet_manager.cc:1950] T f3403116517f45c59f87fa5bda0b4071 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
W20251211 08:58:04.298465 14385 leader_election.cc:343] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.299961 14714 ts_tablet_manager.cc:1434] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.063s user 0.005s sys 0.000s
I20251211 08:58:04.300031 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet c55408f39a414e2caf1db48e6f29eca7 (table default.Table_2a5d1e8d612c4fc5954ea3efc00eb00b [id=ad68b61e866c40f28e5dbd143ccfcb55]) successfully deleted
I20251211 08:58:04.307109 14005 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 2f9c298049734abab7bd934ecf208353
I20251211 08:58:04.308365 14005 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55614 (request call id 657) took 1995 ms. Trace:
I20251211 08:58:04.309129 14414 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f9c298049734abab7bd934ecf208353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:58:03 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:04.307065 15030 raft_consensus.cc:359] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.309712 15030 raft_consensus.cc:385] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:04.309962 15030 raft_consensus.cc:740] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:04.311663 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f9c298049734abab7bd934ecf208353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:58:03 UTC) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:04.315099 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 524) took 3584 ms. Trace:
I20251211 08:58:04.174372 14007 catalog_manager.cc:5654] T 7c0bdd2d0d9f4221bf13812bb7a97025 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } }
I20251211 08:58:04.318867 14904 rpcz_store.cc:276] 1211 08:58:00.731041 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.731151 (+ 110us) service_pool.cc:225] Handling call
1211 08:58:04.137560 (+3406409us) ts_tablet_manager.cc:1151] Deleting tablet f3403116517f45c59f87fa5bda0b4071
1211 08:58:04.137593 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.215208 (+ 77615us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.296632 (+ 81424us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.315037 (+ 18405us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3405908}
I20251211 08:58:04.319861 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_cd6e057b26b443ae9e79158e9350dce5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 3 got 2
I20251211 08:58:04.320132 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 525) took 2251 ms. Trace:
I20251211 08:58:04.320142 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet f3403116517f45c59f87fa5bda0b4071 (table default.Table_6109f16f9985461fbb07b1d083bbb166 [id=58f79d91f6e64d7980b96715d3dd25e7]) successfully deleted
I20251211 08:58:04.319577 14006 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 28f3a31390aa4c4c92dcce796d21f241
I20251211 08:58:04.320487 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:04.320489 14904 rpcz_store.cc:276] 1211 08:58:02.068531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:02.068701 (+ 170us) service_pool.cc:225] Handling call
1211 08:58:04.319823 (+2251122us) ts_tablet_manager.cc:1151] Deleting tablet 65a6feed325f4391a15553e8be914d2d
1211 08:58:04.319860 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.320085 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2250530}
I20251211 08:58:04.308625 14005 rpcz_store.cc:276] 1211 08:58:02.312758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.658842 (+1346084us) service_pool.cc:225] Handling call
1211 08:58:03.660059 (+ 1217us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:58:03.660569 (+ 510us) catalog_manager.cc:2815] Modifying in-memory table state
1211 08:58:03.660667 (+ 98us) catalog_manager.cc:2824] Locking tablets
1211 08:58:03.661134 (+ 467us) catalog_manager.cc:2838] Removing table and tablets from system table
1211 08:58:03.866216 (+205082us) catalog_manager.cc:2856] Removing table from by-name map
1211 08:58:04.306959 (+440743us) catalog_manager.cc:2872] Committing in-memory state
1211 08:58:04.307038 (+ 79us) catalog_manager.cc:2876] Aborting table tasks
1211 08:58:04.308314 (+ 1276us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:03.758783 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:03.759199 (+ 416us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:03.759223 (+ 24us) write_op.cc:435] Acquired schema lock
1211 08:58:03.759234 (+ 11us) tablet.cc:660] Decoding operations
1211 08:58:03.759506 (+ 272us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:03.759544 (+ 38us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:03.759566 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:03.759818 (+ 252us) tablet.cc:699] Row locks acquired
1211 08:58:03.759832 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:58:03.759927 (+ 95us) write_op.cc:270] Start()
1211 08:58:03.760013 (+ 86us) write_op.cc:276] Timestamp: P: 1765443483759965 usec, L: 0
1211 08:58:03.760029 (+ 16us) op_driver.cc:348] REPLICATION: starting
1211 08:58:03.760515 (+ 486us) log.cc:844] Serialized 728 byte log entry
1211 08:58:03.851723 (+ 91208us) op_driver.cc:464] REPLICATION: finished
1211 08:58:03.864169 (+ 12446us) write_op.cc:301] APPLY: starting
1211 08:58:03.864241 (+ 72us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:03.864331 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:03.864423 (+ 92us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:03.864745 (+ 322us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:03.864873 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:58:03.864910 (+ 37us) write_op.cc:312] APPLY: finished
1211 08:58:03.865069 (+ 159us) log.cc:844] Serialized 29 byte log entry
1211 08:58:03.865744 (+ 675us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:03.865828 (+ 84us) write_op.cc:454] Released schema lock
1211 08:58:03.865875 (+ 47us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:04.168095 (+302220us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:04.168141 (+ 46us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":12261,"apply.run_cpu_time_us":1636,"apply.run_wall_time_us":2964,"mutex_wait_us":96087,"num_ops":2,"prepare.queue_time_us":96263,"prepare.run_cpu_time_us":1978,"prepare.run_wall_time_us":2881,"replication_time_us":91607,"thread_start_us":15381,"threads_started":1}]]}
I20251211 08:58:04.322263 14414 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f3a31390aa4c4c92dcce796d21f241 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:58:03 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:04.326957 14005 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "default.Table_210d92e56fb84636a1756a4f3c356e21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:04.328454 14005 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_210d92e56fb84636a1756a4f3c356e21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:04.322031 14266 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f3a31390aa4c4c92dcce796d21f241 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:58:03 UTC) from {username='slave'} at 127.0.0.1:46844
W20251211 08:58:04.329408 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 65a6feed325f4391a15553e8be914d2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65a6feed325f4391a15553e8be914d2d already in progress: creating tablet
I20251211 08:58:04.330802 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 526) took 2231 ms. Trace:
I20251211 08:58:04.332700 14904 rpcz_store.cc:276] 1211 08:58:02.099152 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:02.099292 (+ 140us) service_pool.cc:225] Handling call
1211 08:58:04.330489 (+2231197us) ts_tablet_manager.cc:1151] Deleting tablet a2530df4933249e19d96759910a47673
1211 08:58:04.330526 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.330740 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230609}
I20251211 08:58:04.310482 15030 consensus_queue.cc:260] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:04.340106 14000 catalog_manager.cc:4965] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet a2530df4933249e19d96759910a47673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2530df4933249e19d96759910a47673 already in progress: creating tablet
I20251211 08:58:04.340354 14006 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55614 (request call id 656) took 2031 ms. Trace:
I20251211 08:58:04.343320 14904 tablet_replica.cc:333] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:04.339378 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 4dcd09f7c6364d74bb33503b5bbfea96 (table default.Table_78e567c0deac4f8d99c850bc335bbaf5 [id=482af2b30df84c8891e037e1aea436af]). Expected version 1 got 0
W20251211 08:58:04.365237 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 2bf678f74a12464f9361e379e8af0522 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bf678f74a12464f9361e379e8af0522 already in progress: creating tablet
W20251211 08:58:04.365864 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e967e27dbe1d4ceaae2f1fe5bdacb3a8 already in progress: creating tablet
W20251211 08:58:04.366325 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet f6fb2096f80a43b5a5a6230a824fceb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6fb2096f80a43b5a5a6230a824fceb7 already in progress: creating tablet
I20251211 08:58:04.349800 14288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d8d84e594ea47f3bc9127cc587e8aa7" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:58:04.366833 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet ddbc5b3fad2e43b481ff733b5bad189a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddbc5b3fad2e43b481ff733b5bad189a already in progress: creating tablet
W20251211 08:58:04.367295 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet e9905227c0a844d199362a442e7d630c because the tablet was not found. No further retry: Not found: Tablet not found: e9905227c0a844d199362a442e7d630c
I20251211 08:58:04.368284 14010 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55614 (request call id 659) took 1834 ms. Trace:
I20251211 08:58:04.368589 14010 rpcz_store.cc:276] 1211 08:58:02.534100 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.700276 (+1166176us) service_pool.cc:225] Handling call
1211 08:58:04.304321 (+604045us) catalog_manager.cc:2101] Acquired catalog manager lock
1211 08:58:04.305677 (+ 1356us) catalog_manager.cc:2147] Created new table and tablet info
1211 08:58:04.367727 (+ 62050us) catalog_manager.cc:2207] Wrote table and tablets to system table
1211 08:58:04.367987 (+ 260us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1211 08:58:04.368236 (+ 249us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:04.323710 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:04.324138 (+ 428us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:04.324163 (+ 25us) write_op.cc:435] Acquired schema lock
1211 08:58:04.324179 (+ 16us) tablet.cc:660] Decoding operations
1211 08:58:04.324449 (+ 270us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:04.324502 (+ 53us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:04.324550 (+ 48us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:04.324758 (+ 208us) tablet.cc:699] Row locks acquired
1211 08:58:04.324774 (+ 16us) write_op.cc:260] PREPARE: finished
1211 08:58:04.324884 (+ 110us) write_op.cc:270] Start()
1211 08:58:04.324969 (+ 85us) write_op.cc:276] Timestamp: P: 1765443484324946 usec, L: 0
1211 08:58:04.324987 (+ 18us) op_driver.cc:348] REPLICATION: starting
1211 08:58:04.325487 (+ 500us) log.cc:844] Serialized 433 byte log entry
1211 08:58:04.365869 (+ 40382us) op_driver.cc:464] REPLICATION: finished
1211 08:58:04.366105 (+ 236us) write_op.cc:301] APPLY: starting
1211 08:58:04.366151 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:04.366243 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:04.366320 (+ 77us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:04.366732 (+ 412us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:04.366882 (+ 150us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1211 08:58:04.366924 (+ 42us) write_op.cc:312] APPLY: finished
1211 08:58:04.367076 (+ 152us) log.cc:844] Serialized 29 byte log entry
1211 08:58:04.367197 (+ 121us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:04.367257 (+ 60us) write_op.cc:454] Released schema lock
1211 08:58:04.367284 (+ 27us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:04.367484 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:04.367506 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":82,"apply.run_cpu_time_us":1357,"apply.run_wall_time_us":1357,"num_ops":2,"prepare.queue_time_us":17010,"prepare.run_cpu_time_us":2161,"prepare.run_wall_time_us":39208,"raft.queue_time_us":18237,"raft.run_cpu_time_us":742,"raft.run_wall_time_us":742,"replication_time_us":40786,"thread_start_us":20923,"threads_started":1}]]}
I20251211 08:58:04.346412 14292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "947e6203f910442ca01e2c21587cfe90" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:04.379951 14231 consensus_peers.cc:597] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51c3f5eaef784e868e565f81bd51a5e4. This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:04.380942 14385 leader_election.cc:343] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.378150 14288 raft_consensus.cc:3060] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:04.387161 16373 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4c9cd45be3947d09423ed4508ca2e91)
I20251211 08:58:04.387578 16373 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:58:04.388903 14714 tablet_bootstrap.cc:492] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:04.391549 14103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 596 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:58:04.392633 16320 tablet_replica.cc:333] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:04.394070 14714 tablet_bootstrap.cc:654] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:04.395262 16340 raft_consensus.cc:493] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:04.395691 16340 raft_consensus.cc:515] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:04.397996 15030 ts_tablet_manager.cc:1434] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.126s user 0.006s sys 0.000s
I20251211 08:58:04.397688 16340 leader_election.cc:290] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:04.399276 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6874fb018bc844b3aeeab7db8fc53e9a" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:04.402303 16326 raft_consensus.cc:493] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:04.402712 16326 raft_consensus.cc:515] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.404417 16326 leader_election.cc:290] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:58:04.408169 14542 leader_election.cc:343] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 570c7f08c46e4273bd38c32593dcf304
I20251211 08:58:04.411003 14606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570c7f08c46e4273bd38c32593dcf304" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:04.415345 16373 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:58:04.411484 14606 raft_consensus.cc:2468] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
W20251211 08:58:04.377100 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet f2f095b59ecf44ffabaa0ac22212938b because the tablet was not found. No further retry: Not found: Tablet not found: f2f095b59ecf44ffabaa0ac22212938b
W20251211 08:58:04.423539 13997 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 46475.9us
W20251211 08:58:04.424364 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet da57b72207414512b9744b283bc869fe because the tablet was not found. No further retry: Not found: Tablet not found: da57b72207414512b9744b283bc869fe
W20251211 08:58:04.424949 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 8e05628011634149b17d298c54074d00 because the tablet was not found. No further retry: Not found: Tablet not found: 8e05628011634149b17d298c54074d00
I20251211 08:58:04.425263 14288 raft_consensus.cc:2468] T 5d8d84e594ea47f3bc9127cc587e8aa7 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
W20251211 08:58:04.425486 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet e15483f539f24f1ba1b442663d66fd45 because the tablet was not found. No further retry: Not found: Tablet not found: e15483f539f24f1ba1b442663d66fd45
I20251211 08:58:04.389915 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 596 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:58:04.426218 14385 leader_election.cc:304] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:58:04.426911 16361 raft_consensus.cc:2804] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:04.427037 15758 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:58:04.427316 16361 raft_consensus.cc:697] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:58:04.428244 16373 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251211 08:58:04.427989 16361 consensus_queue.cc:237] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:04.414069 14012 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet c0e0ee99989d41c6af82e330ab8e353c
I20251211 08:58:04.430996 14263 tablet_service.cc:1552] Processing DeleteTablet for tablet c0e0ee99989d41c6af82e330ab8e353c with delete_type TABLET_DATA_DELETED (replaced by tablet e01d89ad1618494fae618112f430a629) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:04.431198 14414 tablet_service.cc:1552] Processing DeleteTablet for tablet c0e0ee99989d41c6af82e330ab8e353c with delete_type TABLET_DATA_DELETED (replaced by tablet e01d89ad1618494fae618112f430a629) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:04.441171 14722 tablet_bootstrap.cc:492] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:04.340690 14006 rpcz_store.cc:276] 1211 08:58:02.308809 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.653800 (+1344991us) service_pool.cc:225] Handling call
1211 08:58:03.654934 (+ 1134us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:58:03.655714 (+ 780us) catalog_manager.cc:2815] Modifying in-memory table state
1211 08:58:03.655856 (+ 142us) catalog_manager.cc:2824] Locking tablets
1211 08:58:03.656321 (+ 465us) catalog_manager.cc:2838] Removing table and tablets from system table
1211 08:58:03.873727 (+217406us) catalog_manager.cc:2856] Removing table from by-name map
1211 08:58:04.319420 (+445693us) catalog_manager.cc:2872] Committing in-memory state
1211 08:58:04.319501 (+ 81us) catalog_manager.cc:2876] Aborting table tasks
1211 08:58:04.340304 (+ 20803us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:03.734952 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:03.735335 (+ 383us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:03.735369 (+ 34us) write_op.cc:435] Acquired schema lock
1211 08:58:03.735387 (+ 18us) tablet.cc:660] Decoding operations
1211 08:58:03.735732 (+ 345us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:03.735778 (+ 46us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:03.735801 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:03.736030 (+ 229us) tablet.cc:699] Row locks acquired
1211 08:58:03.736042 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:58:03.736137 (+ 95us) write_op.cc:270] Start()
1211 08:58:03.736207 (+ 70us) write_op.cc:276] Timestamp: P: 1765443483736167 usec, L: 0
1211 08:58:03.736221 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:58:03.736643 (+ 422us) log.cc:844] Serialized 728 byte log entry
1211 08:58:03.787376 (+ 50733us) op_driver.cc:464] REPLICATION: finished
1211 08:58:03.846902 (+ 59526us) write_op.cc:301] APPLY: starting
1211 08:58:03.846957 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:03.847064 (+ 107us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:03.847160 (+ 96us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:03.847375 (+ 215us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:03.847491 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:58:03.847526 (+ 35us) write_op.cc:312] APPLY: finished
1211 08:58:03.847666 (+ 140us) log.cc:844] Serialized 29 byte log entry
1211 08:58:03.873419 (+ 25753us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:03.873478 (+ 59us) write_op.cc:454] Released schema lock
1211 08:58:03.873506 (+ 28us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:03.901436 (+ 27930us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:03.901463 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":59384,"apply.run_cpu_time_us":1256,"apply.run_wall_time_us":54484,"num_ops":2,"prepare.queue_time_us":77119,"prepare.run_cpu_time_us":1931,"prepare.run_wall_time_us":1930,"raft.queue_time_us":1024,"raft.run_cpu_time_us":546,"raft.run_wall_time_us":546,"replication_time_us":51070,"thread_start_us":63878,"threads_started":1}]]}
I20251211 08:58:04.444255 14722 ts_tablet_manager.cc:1403] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.462s user 0.034s sys 0.013s
W20251211 08:58:04.446627 13999 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2e4c3a63cd7c4cc98a240e459e84a2ac (table default.Table_f9599a5390cc46ecba6001364e0f78bb [id=66855f515acd4e269f106704bf5b6908], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2e4c3a63cd7c4cc98a240e459e84a2ac consensus state has no leader
I20251211 08:58:04.443598 14012 catalog_manager.cc:6724] ReplaceTablet: tablet c0e0ee99989d41c6af82e330ab8e353c deleted and replaced by tablet e01d89ad1618494fae618112f430a629
I20251211 08:58:04.447625 14012 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55614 (request call id 655) took 2368 ms. Trace:
I20251211 08:58:04.450392 14006 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_ee7acb72cac844839994b4a220343f35" } new_table_name: "default.Table_80dcc1d836534f8cbc07d4e17796c655" modify_external_catalogs: true
I20251211 08:58:04.456785 14722 raft_consensus.cc:359] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:04.457324 14722 raft_consensus.cc:385] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:04.457545 14722 raft_consensus.cc:740] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:04.459892 14581 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f3a31390aa4c4c92dcce796d21f241 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:58:03 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.462940 14013 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55614 (request call id 658) took 2029 ms. Trace:
I20251211 08:58:04.463115 16306 raft_consensus.cc:493] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:04.463667 16306 raft_consensus.cc:515] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.465651 14904 raft_consensus.cc:2243] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:04.465742 16306 leader_election.cc:290] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:04.466069 14904 raft_consensus.cc:2272] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:04.466109 14577 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f9c298049734abab7bd934ecf208353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:58:03 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.466956 14585 tablet_service.cc:1552] Processing DeleteTablet for tablet c0e0ee99989d41c6af82e330ab8e353c with delete_type TABLET_DATA_DELETED (replaced by tablet e01d89ad1618494fae618112f430a629) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:04.467679 14288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18036362c93413fa7690906a0298784" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:04.468765 14385 leader_election.cc:343] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.470202 14904 ts_tablet_manager.cc:1916] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:04.477975 16320 raft_consensus.cc:2243] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:04.478541 16320 raft_consensus.cc:2272] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:58:04.478559 14230 leader_election.cc:343] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:58:04.479562 14230 leader_election.cc:343] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.481056 14714 tablet_bootstrap.cc:492] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
W20251211 08:58:04.481272 14382 leader_election.cc:343] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.499488 14382 leader_election.cc:304] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:04.500941 16364 raft_consensus.cc:2749] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:04.493680 16320 ts_tablet_manager.cc:1916] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:04.502959 13896 master-stress-test.cc:577] Killing test thread 1/10
I20251211 08:58:04.507140 14597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6874fb018bc844b3aeeab7db8fc53e9a" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:04.507297 14288 raft_consensus.cc:3060] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:04.447933 14012 rpcz_store.cc:276] 1211 08:58:02.079161 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.650239 (+1571078us) service_pool.cc:225] Handling call
1211 08:58:04.447568 (+797329us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:03.767649 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:03.768029 (+ 380us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:03.768051 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:58:03.768064 (+ 13us) tablet.cc:660] Decoding operations
1211 08:58:03.768340 (+ 276us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:03.768375 (+ 35us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:03.768398 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:03.768574 (+ 176us) tablet.cc:699] Row locks acquired
1211 08:58:03.768588 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:58:03.768680 (+ 92us) write_op.cc:270] Start()
1211 08:58:03.768752 (+ 72us) write_op.cc:276] Timestamp: P: 1765443483768720 usec, L: 0
1211 08:58:03.768768 (+ 16us) op_driver.cc:348] REPLICATION: starting
1211 08:58:03.769232 (+ 464us) log.cc:844] Serialized 659 byte log entry
1211 08:58:03.867485 (+ 98253us) op_driver.cc:464] REPLICATION: finished
1211 08:58:03.901785 (+ 34300us) write_op.cc:301] APPLY: starting
1211 08:58:03.901847 (+ 62us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:03.901934 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:03.902007 (+ 73us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:03.902335 (+ 328us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:03.902516 (+ 181us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:03.902555 (+ 39us) write_op.cc:312] APPLY: finished
1211 08:58:03.902720 (+ 165us) log.cc:844] Serialized 29 byte log entry
1211 08:58:03.903704 (+ 984us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:03.903989 (+ 285us) write_op.cc:454] Released schema lock
1211 08:58:03.904661 (+ 672us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":34146,"apply.run_cpu_time_us":2241,"apply.run_wall_time_us":3840,"num_ops":2,"prepare.queue_time_us":5446,"prepare.run_cpu_time_us":1766,"prepare.run_wall_time_us":2739,"replication_time_us":98606,"thread_start_us":75591,"threads_started":1}]]}
I20251211 08:58:04.508312 16355 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:42846 (request call id 570) took 1200 ms. Trace:
I20251211 08:58:04.512148 16355 rpcz_store.cc:276] 1211 08:58:03.274656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.454585 (+179929us) service_pool.cc:225] Handling call
1211 08:58:04.474892 (+1020307us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:03.490941 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1211 08:58:03.491259 (+ 318us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1211 08:58:03.491297 (+ 38us) alter_schema_op.cc:66] Acquired schema lock
1211 08:58:03.491366 (+ 69us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1211 08:58:03.491478 (+ 112us) alter_schema_op.cc:128] START. Timestamp: 1765443483491679
1211 08:58:03.491502 (+ 24us) op_driver.cc:348] REPLICATION: starting
1211 08:58:03.492039 (+ 537us) log.cc:844] Serialized 173 byte log entry
1211 08:58:04.005337 (+513298us) op_driver.cc:464] REPLICATION: finished
1211 08:58:04.242255 (+236918us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1211 08:58:04.243850 (+ 1595us) log.cc:844] Serialized 5 byte log entry
1211 08:58:04.395149 (+151299us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.395941 (+ 792us) log.cc:844] Serialized 14 byte log entry
1211 08:58:04.428806 (+ 32865us) alter_schema_op.cc:73] Released schema lock
1211 08:58:04.459053 (+ 30247us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":236710,"mutex_wait_us":18251,"prepare.queue_time_us":3172,"prepare.run_cpu_time_us":44024,"prepare.run_wall_time_us":428445,"raft.queue_time_us":33179,"raft.run_cpu_time_us":8839,"raft.run_wall_time_us":249514,"replication_time_us":513743,"thread_start_us":646935,"threads_started":2,"wal-append.queue_time_us":481703}]]}
I20251211 08:58:04.522753 14714 ts_tablet_manager.cc:1403] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.134s user 0.003s sys 0.010s
I20251211 08:58:04.523574 14230 leader_election.cc:304] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:58:04.524685 16096 raft_consensus.cc:2804] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:58:04.524962 16096 raft_consensus.cc:493] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:04.463227 14013 rpcz_store.cc:276] 1211 08:58:02.433337 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.669229 (+1235892us) service_pool.cc:225] Handling call
1211 08:58:04.322812 (+653583us) catalog_manager.cc:2101] Acquired catalog manager lock
1211 08:58:04.324046 (+ 1234us) catalog_manager.cc:2147] Created new table and tablet info
1211 08:58:04.462379 (+138333us) catalog_manager.cc:2207] Wrote table and tablets to system table
1211 08:58:04.462651 (+ 272us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1211 08:58:04.462890 (+ 239us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:04.363131 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:04.367321 (+ 4190us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:04.367356 (+ 35us) write_op.cc:435] Acquired schema lock
1211 08:58:04.367377 (+ 21us) tablet.cc:660] Decoding operations
1211 08:58:04.367725 (+ 348us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:04.367776 (+ 51us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:04.367806 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:04.368058 (+ 252us) tablet.cc:699] Row locks acquired
1211 08:58:04.368089 (+ 31us) write_op.cc:260] PREPARE: finished
1211 08:58:04.368227 (+ 138us) write_op.cc:270] Start()
1211 08:58:04.368321 (+ 94us) write_op.cc:276] Timestamp: P: 1765443484368301 usec, L: 0
1211 08:58:04.368343 (+ 22us) op_driver.cc:348] REPLICATION: starting
1211 08:58:04.368935 (+ 592us) log.cc:844] Serialized 433 byte log entry
1211 08:58:04.449427 (+ 80492us) op_driver.cc:464] REPLICATION: finished
1211 08:58:04.453267 (+ 3840us) write_op.cc:301] APPLY: starting
1211 08:58:04.453312 (+ 45us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:04.453397 (+ 85us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:04.453477 (+ 80us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:04.453903 (+ 426us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:04.454022 (+ 119us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1211 08:58:04.454061 (+ 39us) write_op.cc:312] APPLY: finished
1211 08:58:04.454323 (+ 262us) log.cc:844] Serialized 29 byte log entry
1211 08:58:04.454504 (+ 181us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:04.454564 (+ 60us) write_op.cc:454] Released schema lock
1211 08:58:04.454596 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3710,"apply.run_cpu_time_us":1506,"apply.run_wall_time_us":1505,"num_ops":2,"prepare.queue_time_us":38018,"prepare.run_cpu_time_us":2431,"prepare.run_wall_time_us":6032,"replication_time_us":80982}]]}
I20251211 08:58:04.525223 16096 raft_consensus.cc:3060] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:58:04.528769 14714 raft_consensus.cc:359] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.523170 14010 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "default.Table_cb9f538930f64a52b0f18805535a8fc1" } new_table_name: "default.Table_d458d069d7d54a4f818a89e76f6d658b" modify_external_catalogs: true
I20251211 08:58:04.531559 16096 raft_consensus.cc:515] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.533630 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:04.547374 14012 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:53580 (request call id 1) took 2442 ms. Trace:
I20251211 08:58:04.547698 14012 rpcz_store.cc:276] 1211 08:58:02.104505 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.526479 (+2421974us) service_pool.cc:225] Handling call
1211 08:58:04.547306 (+ 20827us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:58:04.548784 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:04.549317 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_cd6e057b26b443ae9e79158e9350dce5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 3 got 2
I20251211 08:58:04.520442 16339 consensus_queue.cc:1048] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:04.551965 15030 tablet_bootstrap.cc:492] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
W20251211 08:58:04.533524 14544 consensus_peers.cc:597] T e36c6e2868e143be8e7f72307202a268 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:04.557792 14544 leader_election.cc:304] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:58:04.541847 16375 tablet.cc:1722] T 17ab48721a5c48a7b8a5550be658837d P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:58:04.534956 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570c7f08c46e4273bd38c32593dcf304" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:04.561381 14285 raft_consensus.cc:2393] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5e624bcfd154cd586ef1dedb9940063 in current term 2: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
I20251211 08:58:04.566186 14714 raft_consensus.cc:385] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:04.566439 14714 raft_consensus.cc:740] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:04.567004 14714 consensus_queue.cc:260] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.569454 14714 ts_tablet_manager.cc:1434] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.046s user 0.003s sys 0.000s
I20251211 08:58:04.570233 14714 tablet_bootstrap.cc:492] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:58:04.535588 14382 leader_election.cc:343] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.570935 14382 leader_election.cc:304] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:04.571587 16307 raft_consensus.cc:2749] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:58:04.571909 14382 leader_election.cc:343] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.572281 14382 leader_election.cc:304] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:04.572872 16307 raft_consensus.cc:2749] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:58:04.573119 14382 leader_election.cc:343] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.573443 14382 leader_election.cc:304] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:04.573861 16096 leader_election.cc:290] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:04.574035 16307 raft_consensus.cc:2749] T 1065085c701f48c9ad7206c410ce5985 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:58:04.574345 14382 leader_election.cc:343] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.562031 15030 tablet_bootstrap.cc:654] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:58:04.576948 14544 consensus_peers.cc:597] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:04.578403 16301 consensus_queue.cc:1048] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:04.580405 16320 ts_tablet_manager.cc:1929] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:04.562373 16339 raft_consensus.cc:2749] T 1c02cb6ec01041dba118683a57aecde2 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:04.582254 14544 leader_election.cc:304] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:58:04.583221 16315 raft_consensus.cc:3060] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:58:04.588487 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:58:04.589762 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:04.601296 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (table default.Table_ae8d82f4355240188c054f001b2c05a4 [id=f74e074787144a21ad31d4d89cc509c9]). Expected version 2 got 1
I20251211 08:58:04.605966 14004 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:04.612473 16315 raft_consensus.cc:2749] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:58:04.610263 14005 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55614 (request call id 661) took 1970 ms. Trace:
I20251211 08:58:04.613952 14011 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55614 (request call id 660) took 2015 ms. Trace:
I20251211 08:58:04.614329 14011 rpcz_store.cc:276] 1211 08:58:02.598067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.712056 (+1113989us) service_pool.cc:225] Handling call
1211 08:58:04.415021 (+702965us) catalog_manager.cc:2101] Acquired catalog manager lock
1211 08:58:04.432792 (+ 17771us) catalog_manager.cc:2147] Created new table and tablet info
1211 08:58:04.613438 (+180646us) catalog_manager.cc:2207] Wrote table and tablets to system table
1211 08:58:04.613691 (+ 253us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1211 08:58:04.613903 (+ 212us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:04.583171 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:04.584195 (+ 1024us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:04.584223 (+ 28us) write_op.cc:435] Acquired schema lock
1211 08:58:04.584239 (+ 16us) tablet.cc:660] Decoding operations
1211 08:58:04.584574 (+ 335us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:04.584611 (+ 37us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:04.584640 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:04.584839 (+ 199us) tablet.cc:699] Row locks acquired
1211 08:58:04.584855 (+ 16us) write_op.cc:260] PREPARE: finished
1211 08:58:04.584963 (+ 108us) write_op.cc:270] Start()
1211 08:58:04.585047 (+ 84us) write_op.cc:276] Timestamp: P: 1765443484585125 usec, L: 0
1211 08:58:04.585065 (+ 18us) op_driver.cc:348] REPLICATION: starting
1211 08:58:04.607529 (+ 22464us) log.cc:844] Serialized 433 byte log entry
1211 08:58:04.611823 (+ 4294us) op_driver.cc:464] REPLICATION: finished
1211 08:58:04.612012 (+ 189us) write_op.cc:301] APPLY: starting
1211 08:58:04.612056 (+ 44us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:04.612140 (+ 84us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:04.612246 (+ 106us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:04.612621 (+ 375us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:04.612737 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1211 08:58:04.612776 (+ 39us) write_op.cc:312] APPLY: finished
1211 08:58:04.612937 (+ 161us) log.cc:844] Serialized 29 byte log entry
1211 08:58:04.613066 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:04.613123 (+ 57us) write_op.cc:454] Released schema lock
1211 08:58:04.613150 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":68,"apply.run_cpu_time_us":1295,"apply.run_wall_time_us":1296,"num_ops":2,"prepare.queue_time_us":138230,"replication_time_us":26678}]]}
I20251211 08:58:04.613708 14005 rpcz_store.cc:276] 1211 08:58:02.639611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.326605 (+1686994us) service_pool.cc:225] Handling call
1211 08:58:04.328980 (+ 2375us) catalog_manager.cc:2101] Acquired catalog manager lock
1211 08:58:04.330342 (+ 1362us) catalog_manager.cc:2147] Created new table and tablet info
1211 08:58:04.609703 (+279361us) catalog_manager.cc:2207] Wrote table and tablets to system table
1211 08:58:04.609986 (+ 283us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1211 08:58:04.610214 (+ 228us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:04.483350 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:04.483796 (+ 446us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:04.483821 (+ 25us) write_op.cc:435] Acquired schema lock
1211 08:58:04.483831 (+ 10us) tablet.cc:660] Decoding operations
1211 08:58:04.484136 (+ 305us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:04.484175 (+ 39us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:04.484209 (+ 34us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:04.484396 (+ 187us) tablet.cc:699] Row locks acquired
1211 08:58:04.484407 (+ 11us) write_op.cc:260] PREPARE: finished
1211 08:58:04.484521 (+ 114us) write_op.cc:270] Start()
1211 08:58:04.484603 (+ 82us) write_op.cc:276] Timestamp: P: 1765443484484650 usec, L: 0
1211 08:58:04.484619 (+ 16us) op_driver.cc:348] REPLICATION: starting
1211 08:58:04.582102 (+ 97483us) log.cc:844] Serialized 433 byte log entry
1211 08:58:04.582866 (+ 764us) spinlock_profiling.cc:243] Waited 65.1 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbef5a1 00007fc20bc81aa8 00007fc20c0f34d1 00007fc20c0f2ca8 00007fc20c0f7941 00007fc20c0f78f9 00007fc20c0f7889 00007fc20c0f7851 00007fc20c0f6a7d 00000000005aa2b4 00000000005aa0e9
1211 08:58:04.607889 (+ 25023us) op_driver.cc:464] REPLICATION: finished
1211 08:58:04.608149 (+ 260us) write_op.cc:301] APPLY: starting
1211 08:58:04.608208 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:04.608321 (+ 113us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:04.608348 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:04.608810 (+ 462us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:04.608941 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1211 08:58:04.608983 (+ 42us) write_op.cc:312] APPLY: finished
1211 08:58:04.609138 (+ 155us) log.cc:844] Serialized 29 byte log entry
1211 08:58:04.609280 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:04.609342 (+ 62us) write_op.cc:454] Released schema lock
1211 08:58:04.609373 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":1432,"apply.run_wall_time_us":1432,"num_ops":2,"prepare.queue_time_us":68333,"prepare.run_cpu_time_us":2987,"prepare.run_wall_time_us":99686,"raft.queue_time_us":4870,"raft.run_cpu_time_us":861,"raft.run_wall_time_us":15696,"replication_time_us":123157,"spinlock_wait_cycles":150012928}]]}
I20251211 08:58:04.621147 13896 master-stress-test.cc:579] Killed test thread 1/10
I20251211 08:58:04.621380 13896 master-stress-test.cc:577] Killing test thread 2/10
I20251211 08:58:04.621624 13896 master-stress-test.cc:579] Killed test thread 2/10
I20251211 08:58:04.621814 13896 master-stress-test.cc:577] Killing test thread 3/10
I20251211 08:58:04.622027 13896 master-stress-test.cc:579] Killed test thread 3/10
I20251211 08:58:04.622207 13896 master-stress-test.cc:577] Killing test thread 4/10
I20251211 08:58:04.622442 13896 master-stress-test.cc:579] Killed test thread 4/10
I20251211 08:58:04.622649 13896 master-stress-test.cc:577] Killing test thread 5/10
I20251211 08:58:04.623482 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 541) took 4333 ms. Trace:
I20251211 08:58:04.623855 14905 rpcz_store.cc:276] 1211 08:58:00.289797 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.289977 (+ 180us) service_pool.cc:225] Handling call
1211 08:58:04.181924 (+3891947us) ts_tablet_manager.cc:1151] Deleting tablet 51c3f5eaef784e868e565f81bd51a5e4
1211 08:58:04.181969 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.205048 (+ 23079us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.274595 (+ 69547us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.623412 (+348817us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3891326}
I20251211 08:58:04.625010 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 542) took 4333 ms. Trace:
I20251211 08:58:04.625315 14905 rpcz_store.cc:276] 1211 08:58:00.291642 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.291749 (+ 107us) service_pool.cc:225] Handling call
1211 08:58:04.624701 (+4332952us) ts_tablet_manager.cc:1151] Deleting tablet 080d1700df4a4df09acda65c6d021d96
1211 08:58:04.624750 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.624952 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4332496}
I20251211 08:58:04.626196 14905 tablet_replica.cc:333] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:04.627349 14905 raft_consensus.cc:2243] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:04.627820 14905 raft_consensus.cc:2272] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:04.630103 14905 ts_tablet_manager.cc:1916] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:04.631062 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 51c3f5eaef784e868e565f81bd51a5e4 (table default.Table_7343d67071c54141af5a57615f82221e [id=fe5786801bab48888480739b8ff266e8]) successfully deleted
W20251211 08:58:04.631606 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 080d1700df4a4df09acda65c6d021d96 because the tablet was not found. No further retry: Not found: Tablet not found: 080d1700df4a4df09acda65c6d021d96
W20251211 08:58:04.639691 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 57521.3us
I20251211 08:58:04.642599 14230 leader_election.cc:304] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:04.643989 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18036362c93413fa7690906a0298784" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:04.646420 14232 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251211 08:58:04.654647 14722 consensus_queue.cc:260] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:04.656466 15030 tablet_bootstrap.cc:492] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:04.665584 16355 tablet.cc:1722] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251211 08:58:04.682245 14230 consensus_peers.cc:597] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b -> Peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Couldn't send request to peer c5e624bcfd154cd586ef1dedb9940063. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51c3f5eaef784e868e565f81bd51a5e4. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:04.692564 14722 ts_tablet_manager.cc:1434] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.245s user 0.005s sys 0.000s
I20251211 08:58:04.693943 14722 tablet_bootstrap.cc:492] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:04.708813 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570c7f08c46e4273bd38c32593dcf304" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:58:04.709924 15030 ts_tablet_manager.cc:1403] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.158s user 0.004s sys 0.007s
I20251211 08:58:04.574684 14382 leader_election.cc:304] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:04.588045 14414 tablet_service.cc:1552] Processing DeleteTablet for tablet 033c2c016f8f4c01adc8bdbf2885e394 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f0bb5daf5c04698bad02fb5fa2e987e) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:04.599011 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet ed0e90efe651499c929b1fb992712b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:04.615713 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 07f983604bfa474f9c9ce0c982d738fc with delete_type TABLET_DATA_DELETED (replaced by tablet 960bc7a27fa04f2c9f398a67d2c93266) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:04.667214 16369 tablet.cc:1722] T 111adfd1ac1a42bdafa9416594e7e932 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:58:04.714042 16307 raft_consensus.cc:2749] T a74d250ee85a4c768c76a8400003ccdb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:58:04.714213 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 139948us
W20251211 08:58:04.715688 14006 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61775901e28946ddb723f4a01f90feb6 consensus state has no leader
I20251211 08:58:04.718429 16320 log.cc:1199] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/1c02cb6ec01041dba118683a57aecde2
I20251211 08:58:04.724661 14904 ts_tablet_manager.cc:1929] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:04.725535 16320 ts_tablet_manager.cc:1950] T 1c02cb6ec01041dba118683a57aecde2 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
W20251211 08:58:04.726572 14007 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 65) took 2053 ms (client timeout 1000 ms). Trace:
I20251211 08:58:04.732890 14604 raft_consensus.cc:2468] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 2.
W20251211 08:58:04.735786 14382 leader_election.cc:343] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:04.736246 14382 leader_election.cc:304] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:04.736958 16307 raft_consensus.cc:2749] T f18036362c93413fa7690906a0298784 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:04.744319 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 597) took 1464 ms. Trace:
I20251211 08:58:04.744762 16320 rpcz_store.cc:276] 1211 08:58:03.279826 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.280017 (+ 191us) service_pool.cc:225] Handling call
1211 08:58:04.318365 (+1038348us) ts_tablet_manager.cc:1151] Deleting tablet 1c02cb6ec01041dba118683a57aecde2
1211 08:58:04.392495 (+ 74130us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.490545 (+ 98050us) spinlock_profiling.cc:243] Waited 8.16 ms on lock 0x7b2400002018. stack: 00007efeecdf4083 00007efeecdf3f80 00007efeec969574 0000000000508744 00000000005086d9 00007efeecc66781 00007efeecc7a9a4 00007efeecc78aab 00007efeecd079a9 00007efeecd07635 00007efeecd06b90 00007efef3c8bf45 00007efef3ca5979 00007efef3caa113 00007efef42fc445 00007efef4f5331a
1211 08:58:04.524149 (+ 33604us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.580189 (+ 56040us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.744264 (+164075us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":84421,"spinlock_wait_cycles":18773376,"tablet-delete.queue_time_us":1037483}
I20251211 08:58:04.746311 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 1c02cb6ec01041dba118683a57aecde2 (table default.Table_874fb7be95234d289bc2b30e873dabe9 [id=612e052e535b4b34ad98f334e8af58ee]) successfully deleted
I20251211 08:58:04.746611 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 594) took 1475 ms. Trace:
I20251211 08:58:04.745898 15030 raft_consensus.cc:359] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.747267 15030 raft_consensus.cc:385] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:04.747107 16320 rpcz_store.cc:276] 1211 08:58:03.271546 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.281033 (+ 9487us) service_pool.cc:225] Handling call
1211 08:58:04.746371 (+1465338us) ts_tablet_manager.cc:1151] Deleting tablet 570c7f08c46e4273bd38c32593dcf304
1211 08:58:04.746408 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.746565 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1464896}
I20251211 08:58:04.747529 15030 raft_consensus.cc:740] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
W20251211 08:58:04.724905 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 159457us
I20251211 08:58:04.766691 15030 consensus_queue.cc:260] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:04.745553 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 57606.5us
I20251211 08:58:04.754446 16096 raft_consensus.cc:2749] T 6874fb018bc844b3aeeab7db8fc53e9a P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:04.769058 15030 ts_tablet_manager.cc:1434] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.059s user 0.002s sys 0.003s
W20251211 08:58:04.727841 14007 rpcz_store.cc:269] 1211 08:58:02.673619 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.227065 (+553446us) service_pool.cc:225] Handling call
1211 08:58:04.726512 (+1499447us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:04.369274 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:04.369747 (+ 473us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:04.369780 (+ 33us) write_op.cc:435] Acquired schema lock
1211 08:58:04.369796 (+ 16us) tablet.cc:660] Decoding operations
1211 08:58:04.370154 (+ 358us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:04.370186 (+ 32us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:04.370212 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:04.370485 (+ 273us) tablet.cc:699] Row locks acquired
1211 08:58:04.370497 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:58:04.370594 (+ 97us) write_op.cc:270] Start()
1211 08:58:04.370676 (+ 82us) write_op.cc:276] Timestamp: P: 1765443484370676 usec, L: 0
1211 08:58:04.370695 (+ 19us) op_driver.cc:348] REPLICATION: starting
1211 08:58:04.449059 (+ 78364us) log.cc:844] Serialized 907 byte log entry
1211 08:58:04.456532 (+ 7473us) op_driver.cc:464] REPLICATION: finished
1211 08:58:04.475702 (+ 19170us) write_op.cc:301] APPLY: starting
1211 08:58:04.475749 (+ 47us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:04.475841 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:04.475919 (+ 78us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:04.476190 (+ 271us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:04.476311 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:58:04.476359 (+ 48us) write_op.cc:312] APPLY: finished
1211 08:58:04.476511 (+ 152us) log.cc:844] Serialized 29 byte log entry
1211 08:58:04.477340 (+ 829us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:04.477407 (+ 67us) write_op.cc:454] Released schema lock
1211 08:58:04.477437 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":74503,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":19052,"apply.run_cpu_time_us":1242,"apply.run_wall_time_us":1905,"num_ops":2,"prepare.queue_time_us":19162,"prepare.run_cpu_time_us":2309,"prepare.run_wall_time_us":113968,"raft.queue_time_us":153,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":216,"replication_time_us":85761,"thread_start_us":65095,"threads_started":1}]]}
I20251211 08:58:04.783608 16375 tablet.cc:1722] T cb2ff90b54544420b1aad9d2ab4132ff P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:58:04.796473 14905 ts_tablet_manager.cc:1929] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251211 08:58:04.800513 14231 leader_election.cc:343] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: 570c7f08c46e4273bd38c32593dcf304
W20251211 08:58:04.805907 14231 consensus_peers.cc:597] T 1c02cb6ec01041dba118683a57aecde2 P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c02cb6ec01041dba118683a57aecde2. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:04.807019 14905 log.cc:1199] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/ed0e90efe651499c929b1fb992712b96
I20251211 08:58:04.801101 16355 tablet.cc:1722] T c7c04583bef74d84a49cdcd8a1305a88 P c5e624bcfd154cd586ef1dedb9940063: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:58:04.756022 14722 tablet_bootstrap.cc:654] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:04.810554 15030 tablet_bootstrap.cc:492] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:04.810573 14904 log.cc:1199] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/033c2c016f8f4c01adc8bdbf2885e394
I20251211 08:58:04.811281 14904 ts_tablet_manager.cc:1950] T 033c2c016f8f4c01adc8bdbf2885e394 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:04.815714 14905 ts_tablet_manager.cc:1950] T ed0e90efe651499c929b1fb992712b96 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:04.706030 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 7af7c9ab27a64a1681f9874a922e986f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:58:04.711946 16389 log.cc:927] Time spent T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20251211 08:58:04.702873 14714 tablet_bootstrap.cc:654] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
W20251211 08:58:04.819806 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 570c7f08c46e4273bd38c32593dcf304 because the tablet was not found. No further retry: Not found: Tablet not found: 570c7f08c46e4273bd38c32593dcf304
I20251211 08:58:04.820196 16320 tablet_replica.cc:333] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:04.808813 14230 leader_election.cc:304] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters: b276f77f87254a5d909c5ab940a39256
I20251211 08:58:04.822942 16398 tablet.cc:1722] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251211 08:58:04.824857 16315 raft_consensus.cc:493] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:04.825302 16315 raft_consensus.cc:515] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.827024 16315 leader_election.cc:290] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:04.824502 14608 raft_consensus.cc:3060] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:04.839694 16315 consensus_queue.cc:1048] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:04.800173 16369 tablet.cc:1722] T c7c04583bef74d84a49cdcd8a1305a88 P b276f77f87254a5d909c5ab940a39256: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:58:04.827874 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07f983604bfa474f9c9ce0c982d738fc" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:04.843921 14441 raft_consensus.cc:2468] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:04.844914 14542 leader_election.cc:304] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:04.845479 16315 raft_consensus.cc:2804] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:04.845773 16315 raft_consensus.cc:493] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:04.846037 16315 raft_consensus.cc:3060] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:04.846637 15030 tablet_bootstrap.cc:654] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:04.853308 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 543) took 4560 ms. Trace:
I20251211 08:58:04.853466 16320 raft_consensus.cc:2243] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:04.853600 14905 rpcz_store.cc:276] 1211 08:58:00.293169 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:00.293274 (+ 105us) service_pool.cc:225] Handling call
1211 08:58:04.626047 (+4332773us) ts_tablet_manager.cc:1151] Deleting tablet ed0e90efe651499c929b1fb992712b96
1211 08:58:04.626100 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.737667 (+111567us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.796284 (+ 58617us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.853245 (+ 56961us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4332256}
I20251211 08:58:04.854533 16320 raft_consensus.cc:2272] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:04.854715 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet ed0e90efe651499c929b1fb992712b96 (table default.Table_eb3b1d93a3ec4891a9d33b131661fd65 [id=46237116be8e46a68f51e20e008f374b]) successfully deleted
I20251211 08:58:04.856523 16320 ts_tablet_manager.cc:1916] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:04.864534 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 546) took 2793 ms. Trace:
I20251211 08:58:04.864837 14905 rpcz_store.cc:276] 1211 08:58:02.071574 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:02.071689 (+ 115us) service_pool.cc:225] Handling call
1211 08:58:04.863132 (+2791443us) ts_tablet_manager.cc:1151] Deleting tablet 65a6feed325f4391a15553e8be914d2d
1211 08:58:04.863193 (+ 61us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.864470 (+ 1277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2791001}
W20251211 08:58:04.865823 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 65a6feed325f4391a15553e8be914d2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65a6feed325f4391a15553e8be914d2d already in progress: creating tablet
I20251211 08:58:04.854625 16315 raft_consensus.cc:515] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.870756 16315 leader_election.cc:290] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:04.873267 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 547) took 2775 ms. Trace:
I20251211 08:58:04.832472 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 543) took 1760 ms. Trace:
I20251211 08:58:04.874914 14904 rpcz_store.cc:276] 1211 08:58:03.071948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.072075 (+ 127us) service_pool.cc:225] Handling call
1211 08:58:04.343130 (+1271055us) ts_tablet_manager.cc:1151] Deleting tablet 033c2c016f8f4c01adc8bdbf2885e394
1211 08:58:04.343189 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.490178 (+146989us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.724468 (+234290us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:04.832424 (+107956us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":97370,"tablet-delete.queue_time_us":1270550}
I20251211 08:58:04.879475 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 033c2c016f8f4c01adc8bdbf2885e394 (table default.Table_1ce47a525fa9427198a4b8c589d662c7 [id=c7c35cc13c2840fe969a649388e5d250]) successfully deleted
I20251211 08:58:04.884500 14905 rpcz_store.cc:276] 1211 08:58:02.097701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:02.097853 (+ 152us) service_pool.cc:225] Handling call
1211 08:58:04.870660 (+2772807us) ts_tablet_manager.cc:1151] Deleting tablet a2530df4933249e19d96759910a47673
1211 08:58:04.870707 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.873105 (+ 2398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2765883}
W20251211 08:58:04.885792 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet a2530df4933249e19d96759910a47673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2530df4933249e19d96759910a47673 already in progress: creating tablet
I20251211 08:58:04.891439 14905 tablet_replica.cc:333] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:04.892163 14905 raft_consensus.cc:2243] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:04.893229 14905 raft_consensus.cc:2272] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:04.867754 14714 tablet_bootstrap.cc:492] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:04.871559 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07f983604bfa474f9c9ce0c982d738fc" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:04.895017 14440 raft_consensus.cc:3060] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:04.899006 14714 ts_tablet_manager.cc:1403] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.329s user 0.010s sys 0.000s
I20251211 08:58:04.901217 14714 raft_consensus.cc:359] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.901785 14714 raft_consensus.cc:385] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:04.901988 14714 raft_consensus.cc:740] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:04.902704 14714 consensus_queue.cc:260] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.907425 16399 tablet.cc:1722] T c7c04583bef74d84a49cdcd8a1305a88 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251211 08:58:04.910849 16392 consensus_queue.cc:1048] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20251211 08:58:04.824985 16394 raft_consensus.cc:2804] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Leader election won for term 2
I20251211 08:58:04.912192 16394 raft_consensus.cc:697] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 2 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:58:04.913808 16096 consensus_queue.cc:1048] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251211 08:58:04.912977 16394 consensus_queue.cc:237] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.922919 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 554) took 1575 ms. Trace:
I20251211 08:58:04.923219 14904 rpcz_store.cc:276] 1211 08:58:03.347038 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.347148 (+ 110us) service_pool.cc:225] Handling call
1211 08:58:04.875984 (+1528836us) ts_tablet_manager.cc:1151] Deleting tablet 080d1700df4a4df09acda65c6d021d96
1211 08:58:04.922695 (+ 46711us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.922871 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1528355}
W20251211 08:58:04.924225 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet 080d1700df4a4df09acda65c6d021d96 because the tablet was not found. No further retry: Not found: Tablet not found: 080d1700df4a4df09acda65c6d021d96
I20251211 08:58:04.925197 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 555) took 1576 ms. Trace:
I20251211 08:58:04.925513 14904 rpcz_store.cc:276] 1211 08:58:03.348360 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.348582 (+ 222us) service_pool.cc:225] Handling call
1211 08:58:04.924987 (+1576405us) ts_tablet_manager.cc:1151] Deleting tablet c341b762afd8440bb7198ca134c8dffe
1211 08:58:04.925023 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.925153 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1575799}
I20251211 08:58:04.906616 14257 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:46838 (request call id 74) took 1541 ms. Trace:
W20251211 08:58:04.926827 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet c341b762afd8440bb7198ca134c8dffe because the tablet was not found. No further retry: Not found: Tablet not found: c341b762afd8440bb7198ca134c8dffe
I20251211 08:58:04.927088 14257 rpcz_store.cc:276] 1211 08:58:03.365113 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.365295 (+ 182us) service_pool.cc:225] Handling call
1211 08:58:04.454627 (+1089332us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5400181ad4. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f141cc5a1 00007f0f1425cb78 00007f0f1529d20d 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:58:04.878919 (+424292us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x7b54001720d4. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f141cc5a1 00007f0f1425cb78 00007f0f1529d20d 00007f0f13a25104 00007f0f13a25092 00007f0f13a24fc1 00007f0f13a24f3c 00007f0f13a241f2 00007f0f11f6155c 00007f0f11f60996 00007f0f11f6033f
1211 08:58:04.906543 (+ 27624us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":309085824}
I20251211 08:58:04.928009 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 556) took 1575 ms. Trace:
I20251211 08:58:04.930560 14904 rpcz_store.cc:276] 1211 08:58:03.352293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.352399 (+ 106us) service_pool.cc:225] Handling call
1211 08:58:04.927576 (+1575177us) ts_tablet_manager.cc:1151] Deleting tablet dd14de369e084828afb8a924c46c649b
1211 08:58:04.927615 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.927944 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1574662}
W20251211 08:58:04.931756 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet dd14de369e084828afb8a924c46c649b because the tablet was not found. No further retry: Not found: Tablet not found: dd14de369e084828afb8a924c46c649b
I20251211 08:58:04.937565 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 560) took 1516 ms. Trace:
I20251211 08:58:04.947512 15030 tablet_bootstrap.cc:492] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:04.947886 15030 ts_tablet_manager.cc:1403] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.138s user 0.011s sys 0.000s
I20251211 08:58:04.949973 15030 raft_consensus.cc:359] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.950503 15030 raft_consensus.cc:385] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:04.950716 15030 raft_consensus.cc:740] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:04.951588 15030 consensus_queue.cc:260] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.953917 15030 ts_tablet_manager.cc:1434] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251211 08:58:04.955431 14905 ts_tablet_manager.cc:1916] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:04.942031 14722 tablet_bootstrap.cc:492] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:04.964548 14722 ts_tablet_manager.cc:1403] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.271s user 0.008s sys 0.002s
I20251211 08:58:04.966559 14722 raft_consensus.cc:359] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.968356 14722 raft_consensus.cc:385] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:04.968663 14722 raft_consensus.cc:740] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:04.969339 14722 consensus_queue.cc:260] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:04.971414 14722 ts_tablet_manager.cc:1434] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251211 08:58:04.972451 14722 tablet_bootstrap.cc:492] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:04.980235 14904 rpcz_store.cc:276] 1211 08:58:03.421078 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.432575 (+ 11497us) service_pool.cc:225] Handling call
1211 08:58:04.937360 (+1504785us) ts_tablet_manager.cc:1151] Deleting tablet e9905227c0a844d199362a442e7d630c
1211 08:58:04.937395 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:04.937520 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1502384}
W20251211 08:58:04.981415 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet e9905227c0a844d199362a442e7d630c because the tablet was not found. No further retry: Not found: Tablet not found: e9905227c0a844d199362a442e7d630c
I20251211 08:58:04.982209 14904 tablet_replica.cc:333] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:04.993139 14714 ts_tablet_manager.cc:1434] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.094s user 0.004s sys 0.000s
I20251211 08:58:04.994993 16307 raft_consensus.cc:493] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:05.028548 16307 raft_consensus.cc:515] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:05.030339 16307 leader_election.cc:290] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:05.031745 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27cb4228204c4176b1a348960efd5b26" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:05.032929 14385 leader_election.cc:343] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:05.032466 14714 tablet_bootstrap.cc:492] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:05.046931 14905 ts_tablet_manager.cc:1929] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:05.047333 14905 log.cc:1199] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/033c2c016f8f4c01adc8bdbf2885e394
I20251211 08:58:05.048059 14905 ts_tablet_manager.cc:1950] T 033c2c016f8f4c01adc8bdbf2885e394 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:58:05.059816 13902 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 60569us
I20251211 08:58:05.062835 14904 raft_consensus.cc:2243] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 2 LEADER]: Raft consensus shutting down.
I20251211 08:58:05.065061 14904 pending_rounds.cc:70] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
I20251211 08:58:05.065397 14904 pending_rounds.cc:77] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7231256514235129856 op_type: NO_OP noop_request { }
I20251211 08:58:05.065881 14904 raft_consensus.cc:2889] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:58:05.066386 14904 raft_consensus.cc:2272] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:05.067657 14722 tablet_bootstrap.cc:654] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:05.014148 16320 ts_tablet_manager.cc:1929] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:04.914034 14440 raft_consensus.cc:2468] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:05.071830 14542 leader_election.cc:304] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:05.072633 16320 log.cc:1199] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/c112dfcc301648cda6f0f038b1f2af83
I20251211 08:58:05.073493 16320 ts_tablet_manager.cc:1950] T c112dfcc301648cda6f0f038b1f2af83 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:05.087837 14011 master_service.cc:954] ReplaceTablet: received request to replace tablet 53aac024d68c43bca999eec5bd2def69 from {username='slave'} at 127.0.0.1:55614
I20251211 08:58:05.108572 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 596) took 1829 ms. Trace:
I20251211 08:58:05.112869 16320 rpcz_store.cc:276] 1211 08:58:03.279538 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.282916 (+ 3378us) service_pool.cc:225] Handling call
1211 08:58:04.820033 (+1537117us) ts_tablet_manager.cc:1151] Deleting tablet c112dfcc301648cda6f0f038b1f2af83
1211 08:58:04.820073 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.002853 (+182780us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.013910 (+ 11057us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.108511 (+ 94601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1536533}
I20251211 08:58:05.069730 14904 ts_tablet_manager.cc:1916] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:05.101095 16405 raft_consensus.cc:493] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251211 08:58:05.115540 14542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 43809.8us
I20251211 08:58:05.115274 16405 raft_consensus.cc:515] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:05.117210 16405 leader_election.cc:290] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:05.117769 14604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27cb4228204c4176b1a348960efd5b26" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:05.112499 14230 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 263398us
I20251211 08:58:05.120051 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e36c6e2868e143be8e7f72307202a268" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:05.111119 14714 tablet_bootstrap.cc:654] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:05.126540 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07f983604bfa474f9c9ce0c982d738fc" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:05.127125 14285 raft_consensus.cc:2468] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:05.127104 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet c112dfcc301648cda6f0f038b1f2af83 (table default.Table_cfa8f9f640e54badacad03895e72ca57 [id=ffd28388b51247268985b51fdaf33bf0]) successfully deleted
I20251211 08:58:05.127871 16320 tablet_replica.cc:333] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:05.131062 16320 raft_consensus.cc:2243] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:05.134047 16398 tablet.cc:1722] T 111adfd1ac1a42bdafa9416594e7e932 P 31404d5f58e548519239569b150d975b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251211 08:58:05.137030 16320 raft_consensus.cc:2272] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:05.137243 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07f983604bfa474f9c9ce0c982d738fc" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:05.138286 14295 raft_consensus.cc:3060] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:05.139415 14722 tablet_bootstrap.cc:492] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:05.139792 14722 ts_tablet_manager.cc:1403] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.167s user 0.010s sys 0.000s
I20251211 08:58:05.141796 14722 raft_consensus.cc:359] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.142359 14722 raft_consensus.cc:385] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:05.145905 16320 ts_tablet_manager.cc:1916] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:05.142582 14722 raft_consensus.cc:740] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:05.151206 14722 consensus_queue.cc:260] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.152329 16307 raft_consensus.cc:493] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:05.152729 16307 raft_consensus.cc:515] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.154489 16307 leader_election.cc:290] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:05.158506 16339 raft_consensus.cc:493] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:05.160589 14600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143f20f5c37948068ff2fb0d92f0f34b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:05.160725 16339 raft_consensus.cc:515] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.161705 15030 tablet_bootstrap.cc:492] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:05.156055 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143f20f5c37948068ff2fb0d92f0f34b" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:05.163569 14385 leader_election.cc:343] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:05.158025 14295 raft_consensus.cc:2468] T 07f983604bfa474f9c9ce0c982d738fc P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:05.161383 16305 raft_consensus.cc:2804] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:05.165966 14722 ts_tablet_manager.cc:1434] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.026s user 0.004s sys 0.001s
I20251211 08:58:05.176542 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 564) took 1950 ms. Trace:
I20251211 08:58:05.176874 14905 rpcz_store.cc:276] 1211 08:58:03.225694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.314982 (+ 89288us) service_pool.cc:225] Handling call
1211 08:58:04.886544 (+1571562us) ts_tablet_manager.cc:1151] Deleting tablet 033c2c016f8f4c01adc8bdbf2885e394
1211 08:58:04.891297 (+ 4753us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.009234 (+117937us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.046636 (+ 37402us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.176475 (+129839us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42231,"spinlock_wait_cycles":23296,"tablet-delete.queue_time_us":1570808}
I20251211 08:58:05.178239 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 572) took 1131 ms. Trace:
I20251211 08:58:05.179083 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 033c2c016f8f4c01adc8bdbf2885e394 (table default.Table_1ce47a525fa9427198a4b8c589d662c7 [id=c7c35cc13c2840fe969a649388e5d250]) successfully deleted
I20251211 08:58:05.179621 16305 raft_consensus.cc:697] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:05.180442 16305 consensus_queue.cc:237] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.184957 16339 leader_election.cc:290] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:05.186237 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "476eee1606e74832abf928dbaafc844d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:05.178553 14905 rpcz_store.cc:276] 1211 08:58:04.046845 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.076354 (+ 29509us) service_pool.cc:225] Handling call
1211 08:58:05.177966 (+1101612us) ts_tablet_manager.cc:1151] Deleting tablet da52cace2aab4f9dacef18ec5fa56aaf
1211 08:58:05.178008 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.178172 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101000}
W20251211 08:58:05.188568 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet da52cace2aab4f9dacef18ec5fa56aaf because the tablet was not found. No further retry: Not found: Tablet not found: da52cace2aab4f9dacef18ec5fa56aaf
I20251211 08:58:05.197440 14714 tablet_bootstrap.cc:492] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:05.202163 14714 ts_tablet_manager.cc:1403] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.170s user 0.012s sys 0.004s
I20251211 08:58:05.204625 14714 raft_consensus.cc:359] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.205394 14714 raft_consensus.cc:385] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:05.205680 14714 raft_consensus.cc:740] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:05.206741 15030 tablet_bootstrap.cc:654] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:05.216238 16320 ts_tablet_manager.cc:1929] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:05.216624 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 571) took 1170 ms. Trace:
I20251211 08:58:05.217003 14905 rpcz_store.cc:276] 1211 08:58:04.046552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.102388 (+ 55836us) service_pool.cc:225] Handling call
1211 08:58:05.188668 (+1086280us) ts_tablet_manager.cc:1151] Deleting tablet 2a1b011cb7c647f79bb16ce1c92bcff5
1211 08:58:05.188704 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.216566 (+ 27862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085495}
W20251211 08:58:05.218391 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 2a1b011cb7c647f79bb16ce1c92bcff5 because the tablet was not found. No further retry: Not found: Tablet not found: 2a1b011cb7c647f79bb16ce1c92bcff5
I20251211 08:58:05.219491 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 576) took 1082 ms. Trace:
I20251211 08:58:05.219805 14905 rpcz_store.cc:276] 1211 08:58:04.136924 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.143519 (+ 6595us) service_pool.cc:225] Handling call
1211 08:58:05.217903 (+1074384us) ts_tablet_manager.cc:1151] Deleting tablet 07aef1eea7c04d9cbf35c82e45df7a16
1211 08:58:05.217948 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.219435 (+ 1487us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1037037}
I20251211 08:58:05.220822 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 575) took 1084 ms. Trace:
I20251211 08:58:05.221098 14905 rpcz_store.cc:276] 1211 08:58:04.136661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.158428 (+ 21767us) service_pool.cc:225] Handling call
1211 08:58:05.220605 (+1062177us) ts_tablet_manager.cc:1151] Deleting tablet f3403116517f45c59f87fa5bda0b4071
1211 08:58:05.220641 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.220782 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1034987}
I20251211 08:58:05.221946 14905 tablet_replica.cc:333] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:05.206382 14714 consensus_queue.cc:260] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.223043 16320 log.cc:1199] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/ac6be204692a44fab2db551e79237299
W20251211 08:58:05.223057 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 07aef1eea7c04d9cbf35c82e45df7a16 because the tablet was not found. No further retry: Not found: Tablet not found: 07aef1eea7c04d9cbf35c82e45df7a16
I20251211 08:58:05.222666 14905 raft_consensus.cc:2243] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251211 08:58:05.223631 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet f3403116517f45c59f87fa5bda0b4071 because the tablet was not found. No further retry: Not found: Tablet not found: f3403116517f45c59f87fa5bda0b4071
I20251211 08:58:05.223701 14905 raft_consensus.cc:2272] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:05.225030 14714 ts_tablet_manager.cc:1434] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20251211 08:58:05.225593 14905 ts_tablet_manager.cc:1916] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:58:05.226079 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 63042.2us
I20251211 08:58:05.227739 16320 ts_tablet_manager.cc:1950] T ac6be204692a44fab2db551e79237299 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:05.259991 16405 raft_consensus.cc:493] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:05.260470 16405 raft_consensus.cc:515] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.261678 14601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e36c6e2868e143be8e7f72307202a268" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:05.263010 16405 leader_election.cc:290] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:05.264362 15030 tablet_bootstrap.cc:492] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:05.264842 15030 ts_tablet_manager.cc:1403] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.147s user 0.005s sys 0.009s
W20251211 08:58:05.271616 14382 leader_election.cc:343] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:05.272061 14382 leader_election.cc:304] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:05.273068 16307 raft_consensus.cc:2749] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:05.275024 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7a8acb51a2948f4a018787bfaecb694" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:05.275560 14440 raft_consensus.cc:2468] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
W20251211 08:58:05.275727 14382 leader_election.cc:343] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:05.276150 14382 leader_election.cc:304] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:05.276543 14231 leader_election.cc:304] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:58:05.277345 16405 raft_consensus.cc:2804] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:05.277684 16405 raft_consensus.cc:493] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:05.278023 16405 raft_consensus.cc:3060] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:05.278553 16307 raft_consensus.cc:2749] T 27cb4228204c4176b1a348960efd5b26 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:05.283442 14714 tablet_bootstrap.cc:492] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:05.284658 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 599) took 1970 ms. Trace:
I20251211 08:58:05.284981 16320 rpcz_store.cc:276] 1211 08:58:03.313781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.313893 (+ 112us) service_pool.cc:225] Handling call
1211 08:58:05.127720 (+1813827us) ts_tablet_manager.cc:1151] Deleting tablet ac6be204692a44fab2db551e79237299
1211 08:58:05.127762 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.196234 (+ 68472us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.216042 (+ 19808us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.284604 (+ 68562us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1813151}
I20251211 08:58:05.285888 14295 raft_consensus.cc:3060] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:05.286101 16320 tablet_replica.cc:333] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:05.288555 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet ac6be204692a44fab2db551e79237299 (table default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]) successfully deleted
I20251211 08:58:05.290356 16320 raft_consensus.cc:2243] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:05.291527 14722 tablet_bootstrap.cc:492] T 825d8cf806d94702bdd11e0d63ee0c59 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:05.296142 16320 raft_consensus.cc:2272] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:05.296429 16405 raft_consensus.cc:515] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.296777 14905 ts_tablet_manager.cc:1929] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:05.297561 14905 log.cc:1199] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/c112dfcc301648cda6f0f038b1f2af83
I20251211 08:58:05.300578 15030 raft_consensus.cc:359] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.301177 15030 raft_consensus.cc:385] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:05.301448 15030 raft_consensus.cc:740] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:05.301555 16320 ts_tablet_manager.cc:1916] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:05.302016 15030 consensus_queue.cc:260] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.302541 14722 tablet_bootstrap.cc:654] T 825d8cf806d94702bdd11e0d63ee0c59 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:05.305254 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "476eee1606e74832abf928dbaafc844d" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:05.335222 14905 ts_tablet_manager.cc:1950] T c112dfcc301648cda6f0f038b1f2af83 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:05.338702 16305 raft_consensus.cc:493] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:05.339143 16305 raft_consensus.cc:515] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.340940 16305 leader_election.cc:290] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:05.343263 14230 leader_election.cc:304] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:05.343864 16096 raft_consensus.cc:2749] T e36c6e2868e143be8e7f72307202a268 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:05.343990 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825d8cf806d94702bdd11e0d63ee0c59" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:05.344532 14440 raft_consensus.cc:2468] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:05.345463 14542 leader_election.cc:304] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:05.347129 16305 raft_consensus.cc:2804] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:05.348317 16305 raft_consensus.cc:493] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:05.348713 16305 raft_consensus.cc:3060] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:05.356251 14285 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825d8cf806d94702bdd11e0d63ee0c59" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:05.371284 16412 raft_consensus.cc:493] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:05.358090 14544 leader_election.cc:304] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:58:05.371750 16412 raft_consensus.cc:515] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.373618 16315 raft_consensus.cc:2749] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251211 08:58:05.374538 14544 leader_election.cc:343] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:05.376204 16412 leader_election.cc:290] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:05.377466 14608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7a8acb51a2948f4a018787bfaecb694" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:05.377970 14608 raft_consensus.cc:2468] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:58:05.378966 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7a8acb51a2948f4a018787bfaecb694" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:58:05.379451 14603 raft_consensus.cc:3060] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:05.379313 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7a8acb51a2948f4a018787bfaecb694" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:05.380017 14295 raft_consensus.cc:2393] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5e624bcfd154cd586ef1dedb9940063 in current term 1: Already voted for candidate 31404d5f58e548519239569b150d975b in this term.
I20251211 08:58:05.380014 14714 tablet_bootstrap.cc:654] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:05.380734 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7a8acb51a2948f4a018787bfaecb694" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:05.381255 14440 raft_consensus.cc:2468] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:05.382988 14542 leader_election.cc:304] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:05.385159 15030 ts_tablet_manager.cc:1434] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.120s user 0.005s sys 0.003s
I20251211 08:58:05.384943 16405 leader_election.cc:290] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:05.385535 16320 ts_tablet_manager.cc:1929] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:05.385864 16320 log.cc:1199] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/1b2445f66be54d3fbdc3be964b9a9132
I20251211 08:58:05.386080 15030 tablet_bootstrap.cc:492] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:05.386478 16320 ts_tablet_manager.cc:1950] T 1b2445f66be54d3fbdc3be964b9a9132 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:05.387537 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7a8acb51a2948f4a018787bfaecb694" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:05.388051 14440 raft_consensus.cc:3060] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:05.388301 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 601) took 2057 ms. Trace:
I20251211 08:58:05.388584 16320 rpcz_store.cc:276] 1211 08:58:03.330694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.330795 (+ 101us) service_pool.cc:225] Handling call
1211 08:58:05.285932 (+1955137us) ts_tablet_manager.cc:1151] Deleting tablet 1b2445f66be54d3fbdc3be964b9a9132
1211 08:58:05.285982 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.329389 (+ 43407us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.384121 (+ 54732us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.388238 (+ 4117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1954552}
I20251211 08:58:05.376760 16305 raft_consensus.cc:515] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.401266 14440 raft_consensus.cc:2468] T d7a8acb51a2948f4a018787bfaecb694 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:58:05.403769 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 1b2445f66be54d3fbdc3be964b9a9132 (table default.Table_e71106e05fbc47248f2b5f71482c2231 [id=2cc6cd83f4414e91b8359c3b65336061]) successfully deleted
I20251211 08:58:05.403951 14603 raft_consensus.cc:2468] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:58:05.405081 14904 ts_tablet_manager.cc:1929] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251211 08:58:05.405447 16315 raft_consensus.cc:2764] T d7a8acb51a2948f4a018787bfaecb694 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251211 08:58:05.405678 14904 log.cc:1199] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/570c7f08c46e4273bd38c32593dcf304
I20251211 08:58:05.406554 14904 ts_tablet_manager.cc:1950] T 570c7f08c46e4273bd38c32593dcf304 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:05.405058 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 574) took 1357 ms. Trace:
I20251211 08:58:05.407068 14905 rpcz_store.cc:276] 1211 08:58:04.047372 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.135112 (+ 87740us) service_pool.cc:225] Handling call
1211 08:58:05.221790 (+1086678us) ts_tablet_manager.cc:1151] Deleting tablet c112dfcc301648cda6f0f038b1f2af83
1211 08:58:05.221824 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.280255 (+ 58431us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.296535 (+ 16280us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.374919 (+ 78384us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b5800000098. stack: 00007f0c13f80083 00007f0c13f7ff80 00007f0c13af5574 0000000000508744 00000000005086d9 00007f0c13df2781 00007f0c13f74ac1 00007f0c13f750f2 00007f0c1cdc2419 00007f0c1c018108 00007f0c1c0df462 00007f0c1c10ff80 00007f0c1c0f654d 00007f0c1c0f64f9 00007f0c1c0f6489 00007f0c1c0f6451
1211 08:58:05.405009 (+ 30090us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13052,"spinlock_wait_cycles":27645184,"tablet-delete.queue_time_us":1032379}
I20251211 08:58:05.408424 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet c112dfcc301648cda6f0f038b1f2af83 (table default.Table_cfa8f9f640e54badacad03895e72ca57 [id=ffd28388b51247268985b51fdaf33bf0]) successfully deleted
I20251211 08:58:05.408245 14905 tablet_replica.cc:333] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:05.411223 14905 raft_consensus.cc:2243] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:05.411635 14905 raft_consensus.cc:2272] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:05.413200 14905 ts_tablet_manager.cc:1916] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:05.402372 16307 raft_consensus.cc:493] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:05.426314 16307 raft_consensus.cc:515] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.429483 16307 leader_election.cc:290] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:05.431648 16305 leader_election.cc:290] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:05.433694 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825d8cf806d94702bdd11e0d63ee0c59" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:05.434270 14441 raft_consensus.cc:3060] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:05.404651 16320 tablet_replica.cc:333] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:05.443145 16320 raft_consensus.cc:2243] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:05.443751 16320 pending_rounds.cc:70] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:58:05.443943 16320 pending_rounds.cc:77] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256512236204032 op_type: NO_OP noop_request { }
I20251211 08:58:05.444243 16320 raft_consensus.cc:2889] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:58:05.444499 16320 raft_consensus.cc:2272] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:05.446024 16320 ts_tablet_manager.cc:1916] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:05.448573 14905 ts_tablet_manager.cc:1929] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:05.448935 14905 log.cc:1199] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/570c7f08c46e4273bd38c32593dcf304
I20251211 08:58:05.449535 14905 ts_tablet_manager.cc:1950] T 570c7f08c46e4273bd38c32593dcf304 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:05.452638 14441 raft_consensus.cc:2468] T 825d8cf806d94702bdd11e0d63ee0c59 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:05.461712 14542 leader_election.cc:304] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
W20251211 08:58:05.462394 14544 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 59936us
I20251211 08:58:05.408320 14722 tablet_bootstrap.cc:492] T 825d8cf806d94702bdd11e0d63ee0c59 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:05.464747 14722 ts_tablet_manager.cc:1403] T 825d8cf806d94702bdd11e0d63ee0c59 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.173s user 0.015s sys 0.010s
I20251211 08:58:05.466427 16307 raft_consensus.cc:493] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:05.466974 14722 raft_consensus.cc:359] T 825d8cf806d94702bdd11e0d63ee0c59 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.466939 16307 raft_consensus.cc:515] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.467532 14722 raft_consensus.cc:385] T 825d8cf806d94702bdd11e0d63ee0c59 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:05.467806 14722 raft_consensus.cc:740] T 825d8cf806d94702bdd11e0d63ee0c59 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:05.468393 14722 consensus_queue.cc:260] T 825d8cf806d94702bdd11e0d63ee0c59 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.470011 16307 leader_election.cc:290] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:05.470650 14722 ts_tablet_manager.cc:1434] T 825d8cf806d94702bdd11e0d63ee0c59 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251211 08:58:05.471449 14722 tablet_bootstrap.cc:492] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:05.477262 14722 tablet_bootstrap.cc:654] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:05.478884 16312 raft_consensus.cc:2804] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:05.479329 16312 raft_consensus.cc:697] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:05.480063 16312 consensus_queue.cc:237] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.480763 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 561) took 2032 ms. Trace:
I20251211 08:58:05.481033 14904 rpcz_store.cc:276] 1211 08:58:03.448582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.453707 (+ 5125us) service_pool.cc:225] Handling call
1211 08:58:04.982071 (+1528364us) ts_tablet_manager.cc:1151] Deleting tablet 570c7f08c46e4273bd38c32593dcf304
1211 08:58:04.982117 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.064665 (+ 82548us) spinlock_profiling.cc:243] Waited 79.9 ms on lock 0x7b5400181fd4. stack: 00007f0f0d185083 00007f0f0d184f80 00007f0f0ccfa574 0000000000508744 00000000005086d9 00007f0f141cc5a1 00007f0f1426afe2 00007f0f1468d2ec 00007f0f152e431a 00007f0f15314f80 00007f0f152fb54d 00007f0f152fb4f9 00007f0f152fb489 00007f0f152fb451 00007f0f152fa67d 00000000005aa2b4
1211 08:58:05.264975 (+200310us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.404818 (+139843us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.480712 (+ 75894us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":183796096,"tablet-delete.queue_time_us":1527872}
I20251211 08:58:05.482093 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 562) took 1969 ms. Trace:
I20251211 08:58:05.484059 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f37556f36f7e4001b03f44e545a2ba7f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:05.484930 14382 leader_election.cc:343] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:05.484874 14601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c95565ce3954cc598d7cbd45321d7c9" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:05.485859 14382 leader_election.cc:343] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:05.487969 15030 tablet_bootstrap.cc:654] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:05.492895 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 578) took 1355 ms. Trace:
I20251211 08:58:05.493177 14905 rpcz_store.cc:276] 1211 08:58:04.137324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.159909 (+ 22585us) service_pool.cc:225] Handling call
1211 08:58:05.408115 (+1248206us) ts_tablet_manager.cc:1151] Deleting tablet 570c7f08c46e4273bd38c32593dcf304
1211 08:58:05.408150 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.442002 (+ 33852us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.448383 (+ 6381us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.492845 (+ 44462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213735}
I20251211 08:58:05.494182 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 586) took 1269 ms. Trace:
I20251211 08:58:05.494208 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 570c7f08c46e4273bd38c32593dcf304 (table default.Table_f0e893169e89418ab044ae00cdb78176 [id=6ef69beb843946d080b3f6784f0e3af8]) successfully deleted
I20251211 08:58:05.494446 14905 rpcz_store.cc:276] 1211 08:58:04.224757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.225408 (+ 651us) service_pool.cc:225] Handling call
1211 08:58:05.493955 (+1268547us) ts_tablet_manager.cc:1151] Deleting tablet 1b2445f66be54d3fbdc3be964b9a9132
1211 08:58:05.493991 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.494137 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1268045}
I20251211 08:58:05.495307 14905 tablet_replica.cc:333] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:58:05.495508 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 1b2445f66be54d3fbdc3be964b9a9132 because the tablet was not found. No further retry: Not found: Tablet not found: 1b2445f66be54d3fbdc3be964b9a9132
I20251211 08:58:05.495930 14905 raft_consensus.cc:2243] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:05.498381 14905 raft_consensus.cc:2272] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:58:05.502344 14542 consensus_peers.cc:597] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac6be204692a44fab2db551e79237299. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:05.505026 14714 tablet_bootstrap.cc:492] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:05.505020 14905 ts_tablet_manager.cc:1916] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:05.505425 14714 ts_tablet_manager.cc:1403] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.222s user 0.008s sys 0.001s
I20251211 08:58:05.518950 14714 raft_consensus.cc:359] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.519654 14714 raft_consensus.cc:385] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:05.519942 14714 raft_consensus.cc:740] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:05.520637 14714 consensus_queue.cc:260] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.530174 15030 tablet_bootstrap.cc:492] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:05.530553 15030 ts_tablet_manager.cc:1403] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.145s user 0.011s sys 0.003s
I20251211 08:58:05.542078 14905 ts_tablet_manager.cc:1929] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:05.542433 14905 log.cc:1199] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/6874fb018bc844b3aeeab7db8fc53e9a
I20251211 08:58:05.534015 16320 ts_tablet_manager.cc:1929] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:05.543428 16320 log.cc:1199] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/5d8d84e594ea47f3bc9127cc587e8aa7
I20251211 08:58:05.544144 16320 ts_tablet_manager.cc:1950] T 5d8d84e594ea47f3bc9127cc587e8aa7 P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:05.536485 14714 ts_tablet_manager.cc:1434] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.031s user 0.005s sys 0.002s
I20251211 08:58:05.546006 14714 tablet_bootstrap.cc:492] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:05.547217 15030 raft_consensus.cc:359] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.561378 15030 raft_consensus.cc:385] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:05.561607 15030 raft_consensus.cc:740] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:05.571022 15030 consensus_queue.cc:260] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.575397 15030 ts_tablet_manager.cc:1434] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20251211 08:58:05.582127 14905 ts_tablet_manager.cc:1950] T 6874fb018bc844b3aeeab7db8fc53e9a P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:05.590181 15030 tablet_bootstrap.cc:492] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:05.590313 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 78b3d4806f084402a173498454c5520b (DEFAULT_TABLE table=default.Table_7d8b607b4ed9468ab5b3f58936dd0d7f [id=9f6644645d0244bcbf2e97c89be128d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.591233 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78b3d4806f084402a173498454c5520b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.597124 15030 tablet_bootstrap.cc:654] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:05.598631 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 587) took 1373 ms. Trace:
I20251211 08:58:05.598971 14905 rpcz_store.cc:276] 1211 08:58:04.224995 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.226045 (+ 1050us) service_pool.cc:225] Handling call
1211 08:58:05.495194 (+1269149us) ts_tablet_manager.cc:1151] Deleting tablet 6874fb018bc844b3aeeab7db8fc53e9a
1211 08:58:05.495224 (+ 30us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.526509 (+ 31285us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.541889 (+ 15380us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.598578 (+ 56689us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1619,"tablet-delete.queue_time_us":1268697}
I20251211 08:58:05.599710 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 600) took 2269 ms. Trace:
I20251211 08:58:05.600236 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 6874fb018bc844b3aeeab7db8fc53e9a (table default.Table_1eb40c19ca894f47ba26da25462c5c3a [id=62e4ec0c216b41eebee5490eaba6ef5f]) successfully deleted
I20251211 08:58:05.600003 16320 rpcz_store.cc:276] 1211 08:58:03.330426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.330516 (+ 90us) service_pool.cc:225] Handling call
1211 08:58:05.404518 (+2074002us) ts_tablet_manager.cc:1151] Deleting tablet 5d8d84e594ea47f3bc9127cc587e8aa7
1211 08:58:05.404556 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.479649 (+ 75093us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.533826 (+ 54177us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.599650 (+ 65824us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2072889}
I20251211 08:58:05.600935 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 5d8d84e594ea47f3bc9127cc587e8aa7 (table default.Table_70f3ebdd01dc4e75bc85ab29528865fc [id=67d1517b43294838bda420b792edb9c5]) successfully deleted
I20251211 08:58:05.603777 14714 tablet_bootstrap.cc:654] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:05.605278 14011 catalog_manager.cc:5941] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: Sending DeleteTablet for 3 replicas of tablet 53aac024d68c43bca999eec5bd2def69
I20251211 08:58:05.606671 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 53aac024d68c43bca999eec5bd2def69 with delete_type TABLET_DATA_DELETED (replaced by tablet 7225ab3706c6482ab93f8d16fd208c75) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:05.607398 16320 tablet_replica.cc:333] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:05.607683 14011 catalog_manager.cc:6724] ReplaceTablet: tablet 53aac024d68c43bca999eec5bd2def69 deleted and replaced by tablet 7225ab3706c6482ab93f8d16fd208c75
I20251211 08:58:05.608415 14573 tablet_service.cc:1552] Processing DeleteTablet for tablet 53aac024d68c43bca999eec5bd2def69 with delete_type TABLET_DATA_DELETED (replaced by tablet 7225ab3706c6482ab93f8d16fd208c75) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:05.612879 16320 raft_consensus.cc:2243] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:05.613313 16320 raft_consensus.cc:2272] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:05.616218 14413 tablet_service.cc:1505] Processing CreateTablet for tablet 78b3d4806f084402a173498454c5520b (DEFAULT_TABLE table=default.Table_7d8b607b4ed9468ab5b3f58936dd0d7f [id=9f6644645d0244bcbf2e97c89be128d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.617130 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78b3d4806f084402a173498454c5520b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.624087 14573 tablet_service.cc:1505] Processing CreateTablet for tablet 89622b72ab8e4c41822789cdb36a40a9 (DEFAULT_TABLE table=default.Table_b258afac6a8e48bcacfc67b25ed79e41 [id=638b6764db4a44b38d82ecd211499db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.615109 16320 ts_tablet_manager.cc:1916] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:05.631525 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 89622b72ab8e4c41822789cdb36a40a9 (DEFAULT_TABLE table=default.Table_b258afac6a8e48bcacfc67b25ed79e41 [id=638b6764db4a44b38d82ecd211499db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.636111 14230 leader_election.cc:304] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:05.636969 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89622b72ab8e4c41822789cdb36a40a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.645205 14263 tablet_service.cc:1552] Processing DeleteTablet for tablet 53aac024d68c43bca999eec5bd2def69 with delete_type TABLET_DATA_DELETED (replaced by tablet 7225ab3706c6482ab93f8d16fd208c75) from {username='slave'} at 127.0.0.1:46844
I20251211 08:58:05.646885 14266 tablet_service.cc:1505] Processing CreateTablet for tablet 78b3d4806f084402a173498454c5520b (DEFAULT_TABLE table=default.Table_7d8b607b4ed9468ab5b3f58936dd0d7f [id=9f6644645d0244bcbf2e97c89be128d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.647672 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 89622b72ab8e4c41822789cdb36a40a9 (DEFAULT_TABLE table=default.Table_b258afac6a8e48bcacfc67b25ed79e41 [id=638b6764db4a44b38d82ecd211499db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.648728 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78b3d4806f084402a173498454c5520b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.649170 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 570c7f08c46e4273bd38c32593dcf304 (table default.Table_f0e893169e89418ab044ae00cdb78176 [id=6ef69beb843946d080b3f6784f0e3af8]) successfully deleted
I20251211 08:58:05.649931 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89622b72ab8e4c41822789cdb36a40a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.650033 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:05.650731 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (table default.Table_ae8d82f4355240188c054f001b2c05a4 [id=f74e074787144a21ad31d4d89cc509c9]). Expected version 2 got 1
I20251211 08:58:05.651578 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:05.653692 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:05.654436 14008 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:05.656829 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 9761f24cf156422eb87e583b8f73cacb (DEFAULT_TABLE table=default.Table_1b2a75580e22420a81aa5a88d27de913 [id=6095516dfc1444959afee28b76e6455b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.662530 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 583) took 1438 ms. Trace:
I20251211 08:58:05.662837 14905 rpcz_store.cc:276] 1211 08:58:04.223996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.227259 (+ 3263us) service_pool.cc:225] Handling call
1211 08:58:05.599881 (+1372622us) ts_tablet_manager.cc:1151] Deleting tablet f785a3cc2e8145eebae7e9c2885b0b1e
1211 08:58:05.662285 (+ 62404us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.662482 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1372101}
W20251211 08:58:05.663933 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet f785a3cc2e8145eebae7e9c2885b0b1e because the tablet was not found. No further retry: Not found: Tablet not found: f785a3cc2e8145eebae7e9c2885b0b1e
I20251211 08:58:05.666558 14263 tablet_service.cc:1505] Processing CreateTablet for tablet 9761f24cf156422eb87e583b8f73cacb (DEFAULT_TABLE table=default.Table_1b2a75580e22420a81aa5a88d27de913 [id=6095516dfc1444959afee28b76e6455b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.667845 14263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9761f24cf156422eb87e583b8f73cacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.428709 14295 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f37556f36f7e4001b03f44e545a2ba7f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:05.670682 14579 tablet_service.cc:1505] Processing CreateTablet for tablet 259d3b1512e64d2193a0ae59824b2e40 (DEFAULT_TABLE table=default.Table_3ae97e7602364fe19912c59bff54d893 [id=7313c56263b6437d9733d080f0fada1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.674263 14291 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c95565ce3954cc598d7cbd45321d7c9" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:05.675449 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825d8cf806d94702bdd11e0d63ee0c59" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:05.676033 14299 raft_consensus.cc:3060] T 825d8cf806d94702bdd11e0d63ee0c59 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:58:05.676259 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2e4c3a63cd7c4cc98a240e459e84a2ac (table default.Table_f9599a5390cc46ecba6001364e0f78bb [id=66855f515acd4e269f106704bf5b6908], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2e4c3a63cd7c4cc98a240e459e84a2ac consensus state has no leader
I20251211 08:58:05.677650 14279 tablet_service.cc:1505] Processing CreateTablet for tablet 259d3b1512e64d2193a0ae59824b2e40 (DEFAULT_TABLE table=default.Table_3ae97e7602364fe19912c59bff54d893 [id=7313c56263b6437d9733d080f0fada1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.682526 14588 tablet_service.cc:1505] Processing CreateTablet for tablet 395ad0fcfd564b79a10389c8e0a4d69b (DEFAULT_TABLE table=default.Table_9d37990ab8b1472a9b0390875ea8e433 [id=86c9ac6f103a40aa8873c90a4d6f5069]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:58:05.683660 14385 leader_election.cc:343] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:05.684022 14385 leader_election.cc:304] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:05.685292 16307 raft_consensus.cc:2749] T f37556f36f7e4001b03f44e545a2ba7f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:05.688110 14279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259d3b1512e64d2193a0ae59824b2e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.688009 14268 tablet_service.cc:1505] Processing CreateTablet for tablet 395ad0fcfd564b79a10389c8e0a4d69b (DEFAULT_TABLE table=default.Table_9d37990ab8b1472a9b0390875ea8e433 [id=86c9ac6f103a40aa8873c90a4d6f5069]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:58:05.699950 14385 leader_election.cc:343] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:05.700472 14385 leader_election.cc:304] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:05.701045 16307 raft_consensus.cc:2749] T 4c95565ce3954cc598d7cbd45321d7c9 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:05.704349 14414 tablet_service.cc:1505] Processing CreateTablet for tablet 395ad0fcfd564b79a10389c8e0a4d69b (DEFAULT_TABLE table=default.Table_9d37990ab8b1472a9b0390875ea8e433 [id=86c9ac6f103a40aa8873c90a4d6f5069]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.704535 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 259d3b1512e64d2193a0ae59824b2e40 (DEFAULT_TABLE table=default.Table_3ae97e7602364fe19912c59bff54d893 [id=7313c56263b6437d9733d080f0fada1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.705613 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 9761f24cf156422eb87e583b8f73cacb (DEFAULT_TABLE table=default.Table_1b2a75580e22420a81aa5a88d27de913 [id=6095516dfc1444959afee28b76e6455b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.705767 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259d3b1512e64d2193a0ae59824b2e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.706375 14414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 395ad0fcfd564b79a10389c8e0a4d69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.706857 14577 tablet_service.cc:1505] Processing CreateTablet for tablet 26ffb3068e1a407caa55100b34d88b87 (DEFAULT_TABLE table=default.Table_4f8bc056ef464b31b042453e664e7586 [id=a2e0ef8c46d441aa978d5c56ebb22dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.707911 14585 tablet_service.cc:1505] Processing CreateTablet for tablet b3e75c7f2e4a4887a5d0e6806eb9bae6 (DEFAULT_TABLE table=default.Table_36432f79cf0a4a53bdc732d932422846 [id=8d03c5394e2848b29314b6397418ce4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.714938 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9761f24cf156422eb87e583b8f73cacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.715518 14588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 395ad0fcfd564b79a10389c8e0a4d69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.482337 14904 rpcz_store.cc:276] 1211 08:58:03.512475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.513642 (+ 1167us) service_pool.cc:225] Handling call
1211 08:58:05.481873 (+1968231us) ts_tablet_manager.cc:1151] Deleting tablet f785a3cc2e8145eebae7e9c2885b0b1e
1211 08:58:05.481919 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.482050 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1967594}
I20251211 08:58:05.717629 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ffb3068e1a407caa55100b34d88b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.719611 14419 tablet_service.cc:1505] Processing CreateTablet for tablet b3e75c7f2e4a4887a5d0e6806eb9bae6 (DEFAULT_TABLE table=default.Table_36432f79cf0a4a53bdc732d932422846 [id=8d03c5394e2848b29314b6397418ce4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.720595 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3e75c7f2e4a4887a5d0e6806eb9bae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.721657 16312 raft_consensus.cc:493] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:05.722586 14573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89622b72ab8e4c41822789cdb36a40a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.722615 16312 raft_consensus.cc:515] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.728998 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259d3b1512e64d2193a0ae59824b2e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.731083 16312 leader_election.cc:290] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:05.738992 14585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3e75c7f2e4a4887a5d0e6806eb9bae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.750385 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 584) took 1504 ms. Trace:
I20251211 08:58:05.750695 14905 rpcz_store.cc:276] 1211 08:58:04.224247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.228290 (+ 4043us) service_pool.cc:225] Handling call
1211 08:58:05.663604 (+1435314us) ts_tablet_manager.cc:1151] Deleting tablet c341b762afd8440bb7198ca134c8dffe
1211 08:58:05.714846 (+ 51242us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.729058 (+ 14212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1434865}
W20251211 08:58:05.751729 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet c341b762afd8440bb7198ca134c8dffe because the tablet was not found. No further retry: Not found: Tablet not found: c341b762afd8440bb7198ca134c8dffe
I20251211 08:58:05.756217 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 582) took 1532 ms. Trace:
I20251211 08:58:05.756492 14905 rpcz_store.cc:276] 1211 08:58:04.223751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.228851 (+ 5100us) service_pool.cc:225] Handling call
1211 08:58:05.751395 (+1522544us) ts_tablet_manager.cc:1151] Deleting tablet 45c7d841bd3643a799a8b96e82d6d675
1211 08:58:05.755990 (+ 4595us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b5800000098. stack: 00007f0c13f80083 00007f0c13f7ff80 00007f0c13af5574 0000000000508744 00000000005086d9 00007f0c13df2781 00007f0c13f74ac1 00007f0c13f750f2 00007f0c1cdc2419 00007f0c1c018108 00007f0c1c0de9a9 00007f0c1c0defa3 00007f0c1c10ff80 00007f0c1c0f654d 00007f0c1c0f64f9 00007f0c1c0f6489
1211 08:58:05.756035 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.756173 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9259904,"tablet-delete.queue_time_us":1522087}
I20251211 08:58:05.757601 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 590) took 1318 ms. Trace:
W20251211 08:58:05.757579 13997 catalog_manager.cc:4952] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 45c7d841bd3643a799a8b96e82d6d675 because the tablet was not found. No further retry: Not found: Tablet not found: 45c7d841bd3643a799a8b96e82d6d675
I20251211 08:58:05.757865 14905 rpcz_store.cc:276] 1211 08:58:04.439628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.439731 (+ 103us) service_pool.cc:225] Handling call
1211 08:58:05.757265 (+1317534us) ts_tablet_manager.cc:1151] Deleting tablet 28f3a31390aa4c4c92dcce796d21f241
1211 08:58:05.757297 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.757556 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1296980}
W20251211 08:58:05.758955 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 28f3a31390aa4c4c92dcce796d21f241 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28f3a31390aa4c4c92dcce796d21f241 already in progress: creating tablet
I20251211 08:58:05.762605 14440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd09f7c6364d74bb33503b5bbfea96" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:05.762986 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9761f24cf156422eb87e583b8f73cacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.770876 14268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 395ad0fcfd564b79a10389c8e0a4d69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.763249 14440 raft_consensus.cc:2468] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:05.781579 14714 tablet_bootstrap.cc:492] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:05.781953 14714 ts_tablet_manager.cc:1403] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.236s user 0.010s sys 0.000s
W20251211 08:58:05.782157 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet f785a3cc2e8145eebae7e9c2885b0b1e because the tablet was not found. No further retry: Not found: Tablet not found: f785a3cc2e8145eebae7e9c2885b0b1e
I20251211 08:58:05.780423 14542 leader_election.cc:304] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:05.785962 14904 tablet_replica.cc:333] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:05.799576 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 3b4dbde3ecde4b1b8f2b28986ad7eda0 (DEFAULT_TABLE table=default.Table_1e0510e1cd434ce0838cdf8b1915fe66 [id=bdc01a78af4f45209d34377df6f9729e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.801286 14260 tablet_service.cc:1505] Processing CreateTablet for tablet b3e75c7f2e4a4887a5d0e6806eb9bae6 (DEFAULT_TABLE table=default.Table_36432f79cf0a4a53bdc732d932422846 [id=8d03c5394e2848b29314b6397418ce4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.802276 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3e75c7f2e4a4887a5d0e6806eb9bae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.803390 16096 raft_consensus.cc:2804] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:05.804987 14288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd09f7c6364d74bb33503b5bbfea96" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:05.814962 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 589) took 1375 ms. Trace:
I20251211 08:58:05.815264 14905 rpcz_store.cc:276] 1211 08:58:04.439260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.439376 (+ 116us) service_pool.cc:225] Handling call
1211 08:58:05.758610 (+1319234us) ts_tablet_manager.cc:1151] Deleting tablet 2f9c298049734abab7bd934ecf208353
1211 08:58:05.758643 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.814912 (+ 56269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1291026}
W20251211 08:58:05.816365 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 2f9c298049734abab7bd934ecf208353 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f9c298049734abab7bd934ecf208353 already in progress: creating tablet
I20251211 08:58:05.817816 14714 raft_consensus.cc:359] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:05.818399 14714 raft_consensus.cc:385] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:05.818643 14714 raft_consensus.cc:740] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:05.819339 14714 consensus_queue.cc:260] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:05.820475 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 26ffb3068e1a407caa55100b34d88b87 (DEFAULT_TABLE table=default.Table_4f8bc056ef464b31b042453e664e7586 [id=a2e0ef8c46d441aa978d5c56ebb22dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.821075 14413 tablet_service.cc:1505] Processing CreateTablet for tablet b9212aaa10f14f3e9932172413db4683 (DEFAULT_TABLE table=default.Table_20187d2b35014f1fb7329b7771d165f1 [id=9ad95bc453b14edb864d0e4a2f350eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.821360 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ffb3068e1a407caa55100b34d88b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.821597 14419 tablet_service.cc:1505] Processing CreateTablet for tablet 3ab84bae91754a538a197d3f47a23bec (DEFAULT_TABLE table=default.Table_e58edd45723d473d8e60bfb96dada676 [id=01e28e37ce864022a9663c4d5d4e5a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.821986 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9212aaa10f14f3e9932172413db4683. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.821084 14414 tablet_service.cc:1505] Processing CreateTablet for tablet 3b4dbde3ecde4b1b8f2b28986ad7eda0 (DEFAULT_TABLE table=default.Table_1e0510e1cd434ce0838cdf8b1915fe66 [id=bdc01a78af4f45209d34377df6f9729e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.823702 14414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b4dbde3ecde4b1b8f2b28986ad7eda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.827782 14581 tablet_service.cc:1505] Processing CreateTablet for tablet 3ab84bae91754a538a197d3f47a23bec (DEFAULT_TABLE table=default.Table_e58edd45723d473d8e60bfb96dada676 [id=01e28e37ce864022a9663c4d5d4e5a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.828712 14581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ab84bae91754a538a197d3f47a23bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.828651 14276 tablet_service.cc:1505] Processing CreateTablet for tablet 26ffb3068e1a407caa55100b34d88b87 (DEFAULT_TABLE table=default.Table_4f8bc056ef464b31b042453e664e7586 [id=a2e0ef8c46d441aa978d5c56ebb22dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.829658 14276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ffb3068e1a407caa55100b34d88b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.830057 14278 tablet_service.cc:1505] Processing CreateTablet for tablet 3b4dbde3ecde4b1b8f2b28986ad7eda0 (DEFAULT_TABLE table=default.Table_1e0510e1cd434ce0838cdf8b1915fe66 [id=bdc01a78af4f45209d34377df6f9729e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.831037 14278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b4dbde3ecde4b1b8f2b28986ad7eda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.816164 14905 tablet_replica.cc:333] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:05.832840 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ab84bae91754a538a197d3f47a23bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.836158 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 0fb962d4950e45bd8b9ccf9a76274e56 (DEFAULT_TABLE table=default.Table_617b1cf1914744c1bc22abe21985028d [id=62ad0e617da84813abc3c72b9e4cb402]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.848593 14574 tablet_service.cc:1505] Processing CreateTablet for tablet c249d620c5f146a3ab79970be483c287 (DEFAULT_TABLE table=default.Table_a4ec7037d1b24a1492dc74ae4ea83d06 [id=01c48f306e4843e5b9774d542edb9427]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.848698 14582 tablet_service.cc:1505] Processing CreateTablet for tablet 22bf821a074640dcb311ac26279e535c (DEFAULT_TABLE table=default.Table_62edf14ec336499b99d2f34c758edfe4 [id=b88ff8c06bfe4c4a83a0b3b1ee93bcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.850034 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb962d4950e45bd8b9ccf9a76274e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.852990 14905 raft_consensus.cc:2243] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:05.850330 14299 raft_consensus.cc:2468] T 825d8cf806d94702bdd11e0d63ee0c59 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:05.854709 14427 tablet_service.cc:1505] Processing CreateTablet for tablet c249d620c5f146a3ab79970be483c287 (DEFAULT_TABLE table=default.Table_a4ec7037d1b24a1492dc74ae4ea83d06 [id=01c48f306e4843e5b9774d542edb9427]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.863255 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c249d620c5f146a3ab79970be483c287. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:58:05.865087 14544 leader_election.cc:343] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:05.868062 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 3ab84bae91754a538a197d3f47a23bec (DEFAULT_TABLE table=default.Table_e58edd45723d473d8e60bfb96dada676 [id=01e28e37ce864022a9663c4d5d4e5a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.817271 16305 raft_consensus.cc:2804] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:05.868062 14269 tablet_service.cc:1505] Processing CreateTablet for tablet b9212aaa10f14f3e9932172413db4683 (DEFAULT_TABLE table=default.Table_20187d2b35014f1fb7329b7771d165f1 [id=9ad95bc453b14edb864d0e4a2f350eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.876303 14576 tablet_service.cc:1505] Processing CreateTablet for tablet bbc0e436360e4960b65f442a785a0b99 (DEFAULT_TABLE table=default.Table_bacb3872b73a440d89e615fa9dbddfed [id=4a95680ee46b4f1a9cb7887dd10e5ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.878718 16405 raft_consensus.cc:493] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b276f77f87254a5d909c5ab940a39256)
I20251211 08:58:05.879374 16405 raft_consensus.cc:515] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.880750 14722 tablet_bootstrap.cc:492] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:05.881296 14722 ts_tablet_manager.cc:1403] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.410s user 0.016s sys 0.008s
I20251211 08:58:05.881340 16405 leader_election.cc:290] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:05.883373 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 22bf821a074640dcb311ac26279e535c (DEFAULT_TABLE table=default.Table_62edf14ec336499b99d2f34c758edfe4 [id=b88ff8c06bfe4c4a83a0b3b1ee93bcc2]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:58:05.883785 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 46455.8us
I20251211 08:58:05.883685 14273 tablet_service.cc:1505] Processing CreateTablet for tablet c249d620c5f146a3ab79970be483c287 (DEFAULT_TABLE table=default.Table_a4ec7037d1b24a1492dc74ae4ea83d06 [id=01c48f306e4843e5b9774d542edb9427]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.886744 14007 catalog_manager.cc:5654] T e36c6e2868e143be8e7f72307202a268 P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } }
I20251211 08:58:05.887867 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
I20251211 08:58:05.887670 14272 tablet_service.cc:1505] Processing CreateTablet for tablet 0fb962d4950e45bd8b9ccf9a76274e56 (DEFAULT_TABLE table=default.Table_617b1cf1914744c1bc22abe21985028d [id=62ad0e617da84813abc3c72b9e4cb402]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.888631 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:05.895478 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 22bf821a074640dcb311ac26279e535c (DEFAULT_TABLE table=default.Table_62edf14ec336499b99d2f34c758edfe4 [id=b88ff8c06bfe4c4a83a0b3b1ee93bcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.901381 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ab84bae91754a538a197d3f47a23bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.909736 14413 tablet_service.cc:1505] Processing CreateTablet for tablet 0fb962d4950e45bd8b9ccf9a76274e56 (DEFAULT_TABLE table=default.Table_617b1cf1914744c1bc22abe21985028d [id=62ad0e617da84813abc3c72b9e4cb402]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.911279 14587 tablet_service.cc:1505] Processing CreateTablet for tablet b9212aaa10f14f3e9932172413db4683 (DEFAULT_TABLE table=default.Table_20187d2b35014f1fb7329b7771d165f1 [id=9ad95bc453b14edb864d0e4a2f350eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:05.916043 14714 ts_tablet_manager.cc:1434] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.134s user 0.005s sys 0.000s
I20251211 08:58:05.919440 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22bf821a074640dcb311ac26279e535c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.876891 16305 raft_consensus.cc:493] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:05.924705 15030 tablet_bootstrap.cc:492] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:05.925168 15030 ts_tablet_manager.cc:1403] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.335s user 0.009s sys 0.000s
I20251211 08:58:05.923807 16305 raft_consensus.cc:3060] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:05.927902 15030 raft_consensus.cc:359] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.928460 15030 raft_consensus.cc:385] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:05.928684 15030 raft_consensus.cc:740] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:05.929265 15030 consensus_queue.cc:260] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.931200 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b4dbde3ecde4b1b8f2b28986ad7eda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.931408 15030 ts_tablet_manager.cc:1434] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251211 08:58:05.935192 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22bf821a074640dcb311ac26279e535c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.936576 16320 ts_tablet_manager.cc:1929] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:05.936933 16320 log.cc:1199] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/6874fb018bc844b3aeeab7db8fc53e9a
I20251211 08:58:05.937696 16320 ts_tablet_manager.cc:1950] T 6874fb018bc844b3aeeab7db8fc53e9a P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:05.939607 14574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c249d620c5f146a3ab79970be483c287. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:58:05.950970 14382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.3:37291, user_credentials={real_user=slave}} blocked reactor thread for 65845.7us
I20251211 08:58:05.980000 16305 raft_consensus.cc:515] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.983989 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9212aaa10f14f3e9932172413db4683. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.962584 14722 raft_consensus.cc:359] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.994930 14722 raft_consensus.cc:385] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:05.995205 14722 raft_consensus.cc:740] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:05.995775 14722 consensus_queue.cc:260] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:05.997391 14722 ts_tablet_manager.cc:1434] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.116s user 0.001s sys 0.003s
I20251211 08:58:05.999761 14273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c249d620c5f146a3ab79970be483c287. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.003229 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22bf821a074640dcb311ac26279e535c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.989472 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9212aaa10f14f3e9932172413db4683. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.007193 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb962d4950e45bd8b9ccf9a76274e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.014712 16412 raft_consensus.cc:493] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:06.015118 16412 raft_consensus.cc:515] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.017850 16412 leader_election.cc:290] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:06.018997 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc0e436360e4960b65f442a785a0b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:05.990808 15030 tablet_bootstrap.cc:492] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:06.028591 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5884dc6ef814552893cdf9dccfcce19" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:06.029386 14441 raft_consensus.cc:2468] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:06.031729 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb962d4950e45bd8b9ccf9a76274e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.031949 14714 tablet_bootstrap.cc:492] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:06.037561 14714 tablet_bootstrap.cc:654] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.039175 16436 raft_consensus.cc:493] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:06.039661 16436 raft_consensus.cc:515] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.041311 16436 leader_election.cc:290] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:06.042729 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 602) took 2709 ms. Trace:
I20251211 08:58:06.043790 14288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "061e2b441d9c4b03b12d8ed31dce9595" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:06.043519 16320 rpcz_store.cc:276] 1211 08:58:03.333407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.333887 (+ 480us) service_pool.cc:225] Handling call
1211 08:58:05.607235 (+2273348us) ts_tablet_manager.cc:1151] Deleting tablet 6874fb018bc844b3aeeab7db8fc53e9a
1211 08:58:05.607277 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:05.832831 (+225554us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:05.936382 (+103551us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:06.042684 (+106302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1792,"tablet-delete.queue_time_us":2272752}
I20251211 08:58:06.044688 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 6874fb018bc844b3aeeab7db8fc53e9a (table default.Table_1eb40c19ca894f47ba26da25462c5c3a [id=62e4ec0c216b41eebee5490eaba6ef5f]) successfully deleted
I20251211 08:58:06.044911 16320 tablet_replica.cc:333] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:06.045531 16320 raft_consensus.cc:2243] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:06.045979 16320 raft_consensus.cc:2272] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:58:06.046010 14385 leader_election.cc:343] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:06.047670 16320 ts_tablet_manager.cc:1916] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:06.047703 14267 tablet_service.cc:1505] Processing CreateTablet for tablet bbc0e436360e4960b65f442a785a0b99 (DEFAULT_TABLE table=default.Table_bacb3872b73a440d89e615fa9dbddfed [id=4a95680ee46b4f1a9cb7887dd10e5ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.048903 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc0e436360e4960b65f442a785a0b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.051532 14722 tablet_bootstrap.cc:492] T 7af7c9ab27a64a1681f9874a922e986f P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:06.063601 14579 tablet_service.cc:1505] Processing CreateTablet for tablet 9268fa37044c450e8e45cab499e4b3f5 (DEFAULT_TABLE table=default.Table_f5ae9678b61349408c4e7a5624da28ae [id=c724a024aa194ca890eb88042583ae0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.065155 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9268fa37044c450e8e45cab499e4b3f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.066023 14542 leader_election.cc:304] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:06.078238 14714 tablet_bootstrap.cc:492] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:06.078663 14714 ts_tablet_manager.cc:1403] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.000s
I20251211 08:58:06.081573 14905 raft_consensus.cc:2272] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:06.075317 16301 raft_consensus.cc:2804] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:06.087608 16301 raft_consensus.cc:493] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:06.087908 16301 raft_consensus.cc:3060] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:06.091025 14601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "061e2b441d9c4b03b12d8ed31dce9595" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:06.093689 14344 heartbeater.cc:344] Connected to a master server at 127.13.146.61:34977
I20251211 08:58:06.094345 14714 raft_consensus.cc:359] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.094954 14714 raft_consensus.cc:385] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:06.095180 14714 raft_consensus.cc:740] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:06.095715 14714 consensus_queue.cc:260] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:06.096531 14382 leader_election.cc:343] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:06.096906 14382 leader_election.cc:304] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:06.097460 16431 raft_consensus.cc:2749] T 061e2b441d9c4b03b12d8ed31dce9595 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:06.098981 14714 ts_tablet_manager.cc:1434] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20251211 08:58:06.099421 14424 tablet_service.cc:1505] Processing CreateTablet for tablet bbc0e436360e4960b65f442a785a0b99 (DEFAULT_TABLE table=default.Table_bacb3872b73a440d89e615fa9dbddfed [id=4a95680ee46b4f1a9cb7887dd10e5ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.099865 14714 tablet_bootstrap.cc:492] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:06.100317 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc0e436360e4960b65f442a785a0b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.106611 14714 tablet_bootstrap.cc:654] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.114123 14722 tablet_bootstrap.cc:654] T 7af7c9ab27a64a1681f9874a922e986f P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.126524 14288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5884dc6ef814552893cdf9dccfcce19" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:06.127143 14288 raft_consensus.cc:2468] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
W20251211 08:58:06.132736 14231 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.2:36153, user_credentials={real_user=slave}} blocked reactor thread for 285251us
I20251211 08:58:06.136874 15030 tablet_bootstrap.cc:654] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.137689 14904 raft_consensus.cc:2243] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:06.138226 14904 raft_consensus.cc:2272] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:06.139787 16096 raft_consensus.cc:697] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:58:06.140395 14904 ts_tablet_manager.cc:1916] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:06.151434 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd09f7c6364d74bb33503b5bbfea96" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:06.152010 14441 raft_consensus.cc:3060] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:06.155567 16305 leader_election.cc:290] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:06.167326 14441 raft_consensus.cc:2468] T 4dcd09f7c6364d74bb33503b5bbfea96 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:06.168753 14542 leader_election.cc:304] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:06.169831 16305 raft_consensus.cc:2804] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:06.170433 16305 raft_consensus.cc:697] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:06.171738 16305 consensus_queue.cc:237] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.144065 16301 raft_consensus.cc:515] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.178529 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5884dc6ef814552893cdf9dccfcce19" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:06.179075 14449 raft_consensus.cc:3060] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:06.189544 14449 raft_consensus.cc:2468] T b5884dc6ef814552893cdf9dccfcce19 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:06.190493 14542 leader_election.cc:304] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:06.191812 16421 raft_consensus.cc:2804] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:06.192179 16421 raft_consensus.cc:697] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:06.192822 16421 consensus_queue.cc:237] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.200309 14905 ts_tablet_manager.cc:1916] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:06.219127 16301 leader_election.cc:290] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:06.238106 14714 tablet_bootstrap.cc:492] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:06.238565 14714 ts_tablet_manager.cc:1403] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.139s user 0.026s sys 0.054s
I20251211 08:58:06.240842 14714 raft_consensus.cc:359] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:06.241415 14714 raft_consensus.cc:385] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:06.241650 14714 raft_consensus.cc:740] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:06.242185 14714 consensus_queue.cc:260] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:06.244119 14714 ts_tablet_manager.cc:1434] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20251211 08:58:06.250286 14231 leader_election.cc:343] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: ed0e90efe651499c929b1fb992712b96
I20251211 08:58:06.263082 14577 tablet_service.cc:1505] Processing CreateTablet for tablet a213a79ef38e4726ae0a8b8d8f26a06c (DEFAULT_TABLE table=default.Table_fb66cfb783c94088a6a035c6a95b5f43 [id=505f6917b8ec4a23b581f092b089f044]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.263959 14577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a213a79ef38e4726ae0a8b8d8f26a06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.263839 14267 tablet_service.cc:1505] Processing CreateTablet for tablet 9268fa37044c450e8e45cab499e4b3f5 (DEFAULT_TABLE table=default.Table_f5ae9678b61349408c4e7a5624da28ae [id=c724a024aa194ca890eb88042583ae0a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:58:06.266530 14230 leader_election.cc:343] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: ed0e90efe651499c929b1fb992712b96
I20251211 08:58:06.267702 14424 tablet_service.cc:1505] Processing CreateTablet for tablet 9268fa37044c450e8e45cab499e4b3f5 (DEFAULT_TABLE table=default.Table_f5ae9678b61349408c4e7a5624da28ae [id=c724a024aa194ca890eb88042583ae0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.268669 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9268fa37044c450e8e45cab499e4b3f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.269526 15030 tablet_bootstrap.cc:492] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:06.269938 15030 ts_tablet_manager.cc:1403] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.279s user 0.010s sys 0.000s
I20251211 08:58:06.243831 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 680c8f54e75a4291bb4d769d94e21552 (table default.Table_032686fd837d4292bd6c95749e67b5de [id=387069b529984870961fe2e545fbb221]). Expected version 2 got 0
I20251211 08:58:06.278815 14714 tablet_bootstrap.cc:492] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:06.283149 14274 tablet_service.cc:1505] Processing CreateTablet for tablet a213a79ef38e4726ae0a8b8d8f26a06c (DEFAULT_TABLE table=default.Table_fb66cfb783c94088a6a035c6a95b5f43 [id=505f6917b8ec4a23b581f092b089f044]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.267036 14230 leader_election.cc:304] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:06.287289 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9268fa37044c450e8e45cab499e4b3f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.288414 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet d7a8acb51a2948f4a018787bfaecb694 (table default.Table_9e007a9469d54d99b9471cb8e653b620 [id=de3a242b7e6d480ebf509e458ec002c7]). Expected version 1 got 0
I20251211 08:58:06.289389 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a213a79ef38e4726ae0a8b8d8f26a06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.289770 14007 catalog_manager.cc:5654] T 9cc3373962574253868ababbbe78e57d P b276f77f87254a5d909c5ab940a39256 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } }
I20251211 08:58:06.295360 14714 tablet_bootstrap.cc:654] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.299314 16438 raft_consensus.cc:2749] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251211 08:58:06.299988 14231 consensus_peers.cc:597] T 476eee1606e74832abf928dbaafc844d P 31404d5f58e548519239569b150d975b -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:06.271489 16096 consensus_queue.cc:237] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.304723 16320 ts_tablet_manager.cc:1929] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:06.312861 16392 raft_consensus.cc:493] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:06.313294 16392 raft_consensus.cc:515] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.316066 16392 leader_election.cc:290] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:06.314446 14299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd09f7c6364d74bb33503b5bbfea96" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:06.319618 16421 consensus_queue.cc:1048] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:58:06.321436 14231 leader_election.cc:343] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Not found: Tablet not found: f2f095b59ecf44ffabaa0ac22212938b
I20251211 08:58:06.322642 16440 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4c9cd45be3947d09423ed4508ca2e91)
W20251211 08:58:06.324359 14573 consensus_meta.cc:287] T 89622b72ab8e4c41822789cdb36a40a9 P c5e624bcfd154cd586ef1dedb9940063: Time spent flushing consensus metadata: real 0.574s user 0.008s sys 0.001s
I20251211 08:58:06.323186 16440 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:58:06.328478 16440 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:58:06.329466 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 596 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:58:06.334156 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 596 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:58:06.329620 14714 tablet_bootstrap.cc:492] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:06.335588 14714 ts_tablet_manager.cc:1403] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.004s
I20251211 08:58:06.337172 14427 tablet_service.cc:1505] Processing CreateTablet for tablet a213a79ef38e4726ae0a8b8d8f26a06c (DEFAULT_TABLE table=default.Table_fb66cfb783c94088a6a035c6a95b5f43 [id=505f6917b8ec4a23b581f092b089f044]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.338155 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a213a79ef38e4726ae0a8b8d8f26a06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.338079 14714 raft_consensus.cc:359] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:06.338725 14714 raft_consensus.cc:385] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:06.339026 14714 raft_consensus.cc:740] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:06.339602 14714 consensus_queue.cc:260] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:06.340307 16421 consensus_queue.cc:1048] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:06.341183 14288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5884dc6ef814552893cdf9dccfcce19" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:06.341653 14288 raft_consensus.cc:3060] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:06.349195 15030 raft_consensus.cc:359] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.349674 15030 raft_consensus.cc:385] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:06.350118 15030 raft_consensus.cc:740] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:06.350732 15030 consensus_queue.cc:260] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:06.352698 14544 leader_election.cc:343] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:06.353825 16320 log.cc:1199] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/476eee1606e74832abf928dbaafc844d
I20251211 08:58:06.354564 16320 ts_tablet_manager.cc:1950] T 476eee1606e74832abf928dbaafc844d P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:06.356261 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 603) took 3020 ms. Trace:
I20251211 08:58:06.356539 16320 rpcz_store.cc:276] 1211 08:58:03.335493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.338871 (+ 3378us) service_pool.cc:225] Handling call
1211 08:58:06.044779 (+2705908us) ts_tablet_manager.cc:1151] Deleting tablet 476eee1606e74832abf928dbaafc844d
1211 08:58:06.044816 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:06.057199 (+ 12383us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:06.304518 (+247319us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:06.356213 (+ 51695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2705380}
I20251211 08:58:06.357606 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 476eee1606e74832abf928dbaafc844d (table default.Table_5b713a62983c42f4927bf2e96686ced0 [id=74ec01634f3a4dee9a90892189ff6462]) successfully deleted
I20251211 08:58:06.363166 15758 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:58:06.370314 14714 ts_tablet_manager.cc:1434] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.034s user 0.005s sys 0.001s
I20251211 08:58:06.371209 14714 tablet_bootstrap.cc:492] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:58:06.373003 14230 leader_election.cc:343] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Not found: Tablet not found: f2f095b59ecf44ffabaa0ac22212938b
I20251211 08:58:06.373406 14230 leader_election.cc:304] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31404d5f58e548519239569b150d975b; no voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:06.374523 16438 raft_consensus.cc:2749] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:06.375133 16440 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251211 08:58:06.382360 14288 raft_consensus.cc:2468] T b5884dc6ef814552893cdf9dccfcce19 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:06.290683 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:06.386719 14007 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 4dcd09f7c6364d74bb33503b5bbfea96 (table default.Table_78e567c0deac4f8d99c850bc335bbaf5 [id=482af2b30df84c8891e037e1aea436af]). Expected version 1 got 0
I20251211 08:58:06.391043 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 608) took 2082 ms. Trace:
I20251211 08:58:06.391819 15030 ts_tablet_manager.cc:1434] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.122s user 0.007s sys 0.000s
I20251211 08:58:06.392258 16320 rpcz_store.cc:276] 1211 08:58:04.308578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.309014 (+ 436us) service_pool.cc:225] Handling call
1211 08:58:06.390685 (+2081671us) ts_tablet_manager.cc:1151] Deleting tablet 2f9c298049734abab7bd934ecf208353
1211 08:58:06.390727 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:06.390994 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2081171}
W20251211 08:58:06.396338 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 2f9c298049734abab7bd934ecf208353 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f9c298049734abab7bd934ecf208353 already in progress: creating tablet
I20251211 08:58:06.399019 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 609) took 2077 ms. Trace:
I20251211 08:58:06.400239 14260 tablet_service.cc:1505] Processing CreateTablet for tablet 6c98f0fc99294e09b34cbb70afc09364 (DEFAULT_TABLE table=default.Table_cbc846a77e59420a9e6d78771903f231 [id=3991745e76e34ecc8a13da9fd83efe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.401860 14573 tablet_service.cc:1505] Processing CreateTablet for tablet 6c98f0fc99294e09b34cbb70afc09364 (DEFAULT_TABLE table=default.Table_cbc846a77e59420a9e6d78771903f231 [id=3991745e76e34ecc8a13da9fd83efe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.407795 15030 tablet_bootstrap.cc:492] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:06.423113 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c98f0fc99294e09b34cbb70afc09364. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.426538 14714 tablet_bootstrap.cc:654] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.428530 14722 tablet_bootstrap.cc:492] T 7af7c9ab27a64a1681f9874a922e986f P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:06.428995 14722 ts_tablet_manager.cc:1403] T 7af7c9ab27a64a1681f9874a922e986f P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.378s user 0.009s sys 0.002s
I20251211 08:58:06.399474 16320 rpcz_store.cc:276] 1211 08:58:04.321618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.321731 (+ 113us) service_pool.cc:225] Handling call
1211 08:58:06.397210 (+2075479us) ts_tablet_manager.cc:1151] Deleting tablet 28f3a31390aa4c4c92dcce796d21f241
1211 08:58:06.397249 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:06.398955 (+ 1706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2074592}
I20251211 08:58:06.431205 14722 raft_consensus.cc:359] T 7af7c9ab27a64a1681f9874a922e986f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.431789 14722 raft_consensus.cc:385] T 7af7c9ab27a64a1681f9874a922e986f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:06.432050 14722 raft_consensus.cc:740] T 7af7c9ab27a64a1681f9874a922e986f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:06.432519 14419 tablet_service.cc:1505] Processing CreateTablet for tablet 6c98f0fc99294e09b34cbb70afc09364 (DEFAULT_TABLE table=default.Table_cbc846a77e59420a9e6d78771903f231 [id=3991745e76e34ecc8a13da9fd83efe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.432554 14722 consensus_queue.cc:260] T 7af7c9ab27a64a1681f9874a922e986f P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.433369 14269 tablet_service.cc:1505] Processing CreateTablet for tablet 6f0bb5daf5c04698bad02fb5fa2e987e (DEFAULT_TABLE table=default.Table_1ce47a525fa9427198a4b8c589d662c7 [id=c7c35cc13c2840fe969a649388e5d250]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:58:06.433820 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 28f3a31390aa4c4c92dcce796d21f241 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28f3a31390aa4c4c92dcce796d21f241 already in progress: creating tablet
I20251211 08:58:06.434293 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0bb5daf5c04698bad02fb5fa2e987e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.434958 14722 ts_tablet_manager.cc:1434] T 7af7c9ab27a64a1681f9874a922e986f P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251211 08:58:06.439919 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c98f0fc99294e09b34cbb70afc09364. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.449398 16412 consensus_queue.cc:1048] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:06.455677 16320 tablet_replica.cc:333] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:06.456419 16320 raft_consensus.cc:2243] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:06.456929 16320 raft_consensus.cc:2272] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:06.462466 14905 ts_tablet_manager.cc:1929] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:06.462975 14905 log.cc:1199] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/c0e0ee99989d41c6af82e330ab8e353c
I20251211 08:58:06.435774 14722 tablet_bootstrap.cc:492] T 4dcd09f7c6364d74bb33503b5bbfea96 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:06.483292 14905 ts_tablet_manager.cc:1950] T c0e0ee99989d41c6af82e330ab8e353c P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:06.483687 16320 ts_tablet_manager.cc:1916] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:06.497340 14714 tablet_bootstrap.cc:492] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:06.497905 14714 ts_tablet_manager.cc:1403] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.127s user 0.007s sys 0.003s
I20251211 08:58:06.500483 14714 raft_consensus.cc:359] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:06.501546 14904 ts_tablet_manager.cc:1929] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:06.507654 14714 raft_consensus.cc:385] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:06.507978 14714 raft_consensus.cc:740] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:06.515007 14714 consensus_queue.cc:260] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:06.530124 14714 ts_tablet_manager.cc:1434] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20251211 08:58:06.530989 14714 tablet_bootstrap.cc:492] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:06.543840 16436 raft_consensus.cc:493] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:06.544248 16436 raft_consensus.cc:515] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:06.546080 16436 leader_election.cc:290] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:58:06.548431 16452 log.cc:927] Time spent T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251211 08:58:06.548705 14419 tablet_service.cc:1505] Processing CreateTablet for tablet 6f0bb5daf5c04698bad02fb5fa2e987e (DEFAULT_TABLE table=default.Table_1ce47a525fa9427198a4b8c589d662c7 [id=c7c35cc13c2840fe969a649388e5d250]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.550318 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74c8dd2b036c42a0bec79d3637f00ef4" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:06.554409 14382 leader_election.cc:343] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:06.564229 16431 raft_consensus.cc:493] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:06.564765 16431 raft_consensus.cc:515] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:06.567312 14904 log.cc:1199] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/c112dfcc301648cda6f0f038b1f2af83
I20251211 08:58:06.566648 16431 leader_election.cc:290] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:06.569140 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0bb5daf5c04698bad02fb5fa2e987e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.571250 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f56fd93aeac6435ea84388ce05580113" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:06.573633 14904 ts_tablet_manager.cc:1950] T c112dfcc301648cda6f0f038b1f2af83 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:06.544025 14722 tablet_bootstrap.cc:654] T 4dcd09f7c6364d74bb33503b5bbfea96 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.572075 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74c8dd2b036c42a0bec79d3637f00ef4" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:06.572584 14291 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f56fd93aeac6435ea84388ce05580113" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:06.575610 14385 leader_election.cc:343] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:06.576042 14385 leader_election.cc:304] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
W20251211 08:58:06.576418 14382 leader_election.cc:343] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
W20251211 08:58:06.576956 14385 leader_election.cc:343] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:06.578256 14385 leader_election.cc:304] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:06.578919 16436 raft_consensus.cc:2749] T 74c8dd2b036c42a0bec79d3637f00ef4 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:06.580133 16436 raft_consensus.cc:2749] T f56fd93aeac6435ea84388ce05580113 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:06.581549 14573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c98f0fc99294e09b34cbb70afc09364. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.582407 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 6f0bb5daf5c04698bad02fb5fa2e987e (DEFAULT_TABLE table=default.Table_1ce47a525fa9427198a4b8c589d662c7 [id=c7c35cc13c2840fe969a649388e5d250]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.583324 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0bb5daf5c04698bad02fb5fa2e987e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.588755 14905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42846 (request call id 591) took 2128 ms. Trace:
I20251211 08:58:06.589100 14905 rpcz_store.cc:276] 1211 08:58:04.460847 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.460965 (+ 118us) service_pool.cc:225] Handling call
1211 08:58:05.816016 (+1355051us) ts_tablet_manager.cc:1151] Deleting tablet c0e0ee99989d41c6af82e330ab8e353c
1211 08:58:05.816056 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:06.360786 (+544730us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:06.462280 (+101494us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:06.588701 (+126421us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":303,"tablet-delete.queue_time_us":1348674}
I20251211 08:58:06.589977 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 47c4737ceb504d5f8e4d319566e9f929 (DEFAULT_TABLE table=default.Table_74ae14e8cc744925a5f481dcf5eac8ab [id=9586d40d60eb45b49bcb7649952f2cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.592108 14576 tablet_service.cc:1505] Processing CreateTablet for tablet 47c4737ceb504d5f8e4d319566e9f929 (DEFAULT_TABLE table=default.Table_74ae14e8cc744925a5f481dcf5eac8ab [id=9586d40d60eb45b49bcb7649952f2cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.592978 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c4737ceb504d5f8e4d319566e9f929. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.595790 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet c0e0ee99989d41c6af82e330ab8e353c (table default.Table_7ca1c0fc20ca4dd7a11793c0ac968666 [id=67ab831326db412fa228aac2683950ee]) successfully deleted
W20251211 08:58:06.596449 13997 catalog_manager.cc:4965] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): delete failed for tablet 53aac024d68c43bca999eec5bd2def69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53aac024d68c43bca999eec5bd2def69 already in progress: creating tablet
I20251211 08:58:06.610661 14714 tablet_bootstrap.cc:654] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.611831 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 563) took 3081 ms. Trace:
I20251211 08:58:06.612190 14904 rpcz_store.cc:276] 1211 08:58:03.529965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.530118 (+ 153us) service_pool.cc:225] Handling call
1211 08:58:05.785597 (+2255479us) ts_tablet_manager.cc:1151] Deleting tablet c112dfcc301648cda6f0f038b1f2af83
1211 08:58:05.785679 (+ 82us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:06.415129 (+629450us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:06.501351 (+ 86222us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:06.611767 (+110416us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":350821,"tablet-delete.queue_time_us":2254784}
I20251211 08:58:06.613476 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet c112dfcc301648cda6f0f038b1f2af83 (table default.Table_cfa8f9f640e54badacad03895e72ca57 [id=ffd28388b51247268985b51fdaf33bf0]) successfully deleted
I20251211 08:58:06.613694 14904 tablet_replica.cc:333] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:06.614430 14904 raft_consensus.cc:2243] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:06.614890 14904 raft_consensus.cc:2272] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:06.615278 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c4737ceb504d5f8e4d319566e9f929. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.616804 14904 ts_tablet_manager.cc:1916] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:06.627703 14427 tablet_service.cc:1505] Processing CreateTablet for tablet 47c4737ceb504d5f8e4d319566e9f929 (DEFAULT_TABLE table=default.Table_74ae14e8cc744925a5f481dcf5eac8ab [id=9586d40d60eb45b49bcb7649952f2cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.641013 14574 tablet_service.cc:1505] Processing CreateTablet for tablet aadc0c34b3e14540afc1e4088eb1c913 (DEFAULT_TABLE table=default.Table_b52de3809ace45cb9d262625fc5fb7ce [id=ebed604b3fc24d218d211398c539a32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.642079 14574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadc0c34b3e14540afc1e4088eb1c913. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.650293 14269 tablet_service.cc:1505] Processing CreateTablet for tablet aadc0c34b3e14540afc1e4088eb1c913 (DEFAULT_TABLE table=default.Table_b52de3809ace45cb9d262625fc5fb7ce [id=ebed604b3fc24d218d211398c539a32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.651515 14269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadc0c34b3e14540afc1e4088eb1c913. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.654305 16431 raft_consensus.cc:493] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:06.654742 16431 raft_consensus.cc:515] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.656957 16431 leader_election.cc:290] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:06.658489 14587 tablet_service.cc:1505] Processing CreateTablet for tablet de3681ffc27749629daeb9833fd09bf0 (DEFAULT_TABLE table=default.Table_14d314e70b024f78ad6b5d01e93fc113 [id=793d17b63d9849658ce1e82054c11720]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.658994 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7af7c9ab27a64a1681f9874a922e986f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:06.659440 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de3681ffc27749629daeb9833fd09bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.659528 14603 raft_consensus.cc:2468] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:58:06.660384 14382 leader_election.cc:304] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:06.661513 14413 tablet_service.cc:1505] Processing CreateTablet for tablet aadc0c34b3e14540afc1e4088eb1c913 (DEFAULT_TABLE table=default.Table_b52de3809ace45cb9d262625fc5fb7ce [id=ebed604b3fc24d218d211398c539a32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.661695 16431 raft_consensus.cc:493] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:06.662017 16431 raft_consensus.cc:515] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:06.662438 16436 raft_consensus.cc:2804] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:06.664584 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15595c5a432047de873c2647b78838d6" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:06.662726 16436 raft_consensus.cc:493] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:06.665231 16436 raft_consensus.cc:3060] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:58:06.665576 14382 leader_election.cc:343] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:06.666790 16431 leader_election.cc:290] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:06.667142 16431 raft_consensus.cc:493] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:06.667481 16431 raft_consensus.cc:515] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.669893 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c03f976b926a4a6fba8b70e9df9a4160" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:06.670941 14382 leader_election.cc:343] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:06.674371 15030 tablet_bootstrap.cc:700] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063: Time spent opening tablet: real 0.136s user 0.002s sys 0.001s
I20251211 08:58:06.674937 15030 tablet_bootstrap.cc:654] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.676910 16436 raft_consensus.cc:515] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.679045 16436 leader_election.cc:290] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:06.679741 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7af7c9ab27a64a1681f9874a922e986f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
I20251211 08:58:06.680320 14603 raft_consensus.cc:3060] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:06.694491 16320 ts_tablet_manager.cc:1929] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:06.672013 14287 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7af7c9ab27a64a1681f9874a922e986f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:06.697260 16190 log.cc:927] Time spent T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20251211 08:58:06.697176 14291 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15595c5a432047de873c2647b78838d6" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:06.698346 14293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c03f976b926a4a6fba8b70e9df9a4160" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
I20251211 08:58:06.698452 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7af7c9ab27a64a1681f9874a922e986f" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
I20251211 08:58:06.702466 16312 consensus_queue.cc:1048] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:06.708315 14722 tablet_bootstrap.cc:492] T 4dcd09f7c6364d74bb33503b5bbfea96 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:06.708748 14722 ts_tablet_manager.cc:1403] T 4dcd09f7c6364d74bb33503b5bbfea96 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.273s user 0.009s sys 0.014s
I20251211 08:58:06.711159 14722 raft_consensus.cc:359] T 4dcd09f7c6364d74bb33503b5bbfea96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.718876 14722 raft_consensus.cc:385] T 4dcd09f7c6364d74bb33503b5bbfea96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:06.724133 14722 raft_consensus.cc:740] T 4dcd09f7c6364d74bb33503b5bbfea96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:06.724835 14722 consensus_queue.cc:260] T 4dcd09f7c6364d74bb33503b5bbfea96 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.695879 14287 raft_consensus.cc:2468] T 7af7c9ab27a64a1681f9874a922e986f P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 0.
I20251211 08:58:06.726326 14904 ts_tablet_manager.cc:1929] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:06.731472 14904 log.cc:1199] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/ed0e90efe651499c929b1fb992712b96
I20251211 08:58:06.732231 14904 ts_tablet_manager.cc:1950] T ed0e90efe651499c929b1fb992712b96 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:06.715029 14260 tablet_service.cc:1505] Processing CreateTablet for tablet de3681ffc27749629daeb9833fd09bf0 (DEFAULT_TABLE table=default.Table_14d314e70b024f78ad6b5d01e93fc113 [id=793d17b63d9849658ce1e82054c11720]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.735688 14260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de3681ffc27749629daeb9833fd09bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.740864 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 0f1a55d787fc4bbdbe4858787d7df8c9 (DEFAULT_TABLE table=default.Table_184dbe4f30324cfcbb04f5031169242e [id=6dd6ac573e82475e82f7c9512616261c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.741824 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f1a55d787fc4bbdbe4858787d7df8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.743285 14603 raft_consensus.cc:2468] T 7af7c9ab27a64a1681f9874a922e986f P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276f77f87254a5d909c5ab940a39256 in term 1.
W20251211 08:58:06.699896 14385 leader_election.cc:343] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:06.700120 14414 tablet_service.cc:1505] Processing CreateTablet for tablet de3681ffc27749629daeb9833fd09bf0 (DEFAULT_TABLE table=default.Table_14d314e70b024f78ad6b5d01e93fc113 [id=793d17b63d9849658ce1e82054c11720]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.754626 15030 tablet_bootstrap.cc:492] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:06.758585 15030 ts_tablet_manager.cc:1403] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.351s user 0.005s sys 0.005s
I20251211 08:58:06.758472 14385 leader_election.cc:304] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
W20251211 08:58:06.759846 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 60005.4us
I20251211 08:58:06.760838 15030 raft_consensus.cc:359] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.761013 16436 raft_consensus.cc:2749] T 15595c5a432047de873c2647b78838d6 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:06.761430 15030 raft_consensus.cc:385] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:06.761667 15030 raft_consensus.cc:740] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:06.762571 16320 log.cc:1199] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/c0e0ee99989d41c6af82e330ab8e353c
I20251211 08:58:06.766494 15030 consensus_queue.cc:260] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.726495 14714 tablet_bootstrap.cc:492] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:06.729650 16431 leader_election.cc:290] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:06.750702 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadc0c34b3e14540afc1e4088eb1c913. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.768980 14714 ts_tablet_manager.cc:1403] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.238s user 0.012s sys 0.000s
I20251211 08:58:06.755111 14382 leader_election.cc:304] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:06.775290 14414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de3681ffc27749629daeb9833fd09bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.777091 14275 tablet_service.cc:1505] Processing CreateTablet for tablet 0f1a55d787fc4bbdbe4858787d7df8c9 (DEFAULT_TABLE table=default.Table_184dbe4f30324cfcbb04f5031169242e [id=6dd6ac573e82475e82f7c9512616261c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.778676 16462 raft_consensus.cc:2804] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:06.779132 16462 raft_consensus.cc:697] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Becoming Leader. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Running, Role: LEADER
I20251211 08:58:06.779774 16462 consensus_queue.cc:237] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
W20251211 08:58:06.787483 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 827b6a73ea5a4462ae58f36f12919b0c (table default.Table_6f0fe9f7f8ca4ee69de561cba58b14d4 [id=e777b0c9fe014c3f94da539cdf1196a7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 827b6a73ea5a4462ae58f36f12919b0c consensus state has no leader
I20251211 08:58:06.788357 15030 ts_tablet_manager.cc:1434] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20251211 08:58:06.788882 14427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c4737ceb504d5f8e4d319566e9f929. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.789191 15030 tablet_bootstrap.cc:492] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:06.771042 14714 raft_consensus.cc:359] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:06.803233 14714 raft_consensus.cc:385] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:06.803553 14714 raft_consensus.cc:740] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:06.806021 14580 tablet_service.cc:1505] Processing CreateTablet for tablet a30d6f6f3224444880098beb77a14531 (DEFAULT_TABLE table=default.Table_7ed565fc5cb943fb85b6cf27e3c57e6c [id=5dae6ccab28e4a0ea61c7058328c96bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.809687 14267 tablet_service.cc:1505] Processing CreateTablet for tablet a30d6f6f3224444880098beb77a14531 (DEFAULT_TABLE table=default.Table_7ed565fc5cb943fb85b6cf27e3c57e6c [id=5dae6ccab28e4a0ea61c7058328c96bd]), partition=RANGE (key) PARTITION UNBOUNDED
W20251211 08:58:06.810985 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 50316.3us
W20251211 08:58:06.811565 14385 leader_election.cc:343] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:06.811874 14385 leader_election.cc:304] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:06.812909 16436 raft_consensus.cc:2749] T c03f976b926a4a6fba8b70e9df9a4160 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:06.817041 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30d6f6f3224444880098beb77a14531. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.823894 14714 consensus_queue.cc:260] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:06.832945 14714 ts_tablet_manager.cc:1434] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.064s user 0.005s sys 0.000s
I20251211 08:58:06.834319 14714 tablet_bootstrap.cc:492] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:06.838653 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 565) took 3192 ms. Trace:
I20251211 08:58:06.839017 14904 rpcz_store.cc:276] 1211 08:58:03.646222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.646313 (+ 91us) service_pool.cc:225] Handling call
1211 08:58:06.613540 (+2967227us) ts_tablet_manager.cc:1151] Deleting tablet ed0e90efe651499c929b1fb992712b96
1211 08:58:06.613595 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:06.668269 (+ 54674us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:06.726126 (+ 57857us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:06.838593 (+112467us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2966722}
I20251211 08:58:06.840214 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet ed0e90efe651499c929b1fb992712b96 (table default.Table_eb3b1d93a3ec4891a9d33b131661fd65 [id=46237116be8e46a68f51e20e008f374b]) successfully deleted
I20251211 08:58:06.840678 14722 ts_tablet_manager.cc:1434] T 4dcd09f7c6364d74bb33503b5bbfea96 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.132s user 0.005s sys 0.000s
I20251211 08:58:06.843623 14267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30d6f6f3224444880098beb77a14531. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.847203 14275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f1a55d787fc4bbdbe4858787d7df8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.847349 14274 tablet_service.cc:1505] Processing CreateTablet for tablet 8b9ba7e0592347ce80e47d461ad1a4e2 (DEFAULT_TABLE table=default.Table_df875ee987f64781b50ea89729e90a02 [id=c6c7f21718f645d5a26d3f8821ba470b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.847427 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 8b9ba7e0592347ce80e47d461ad1a4e2 (DEFAULT_TABLE table=default.Table_df875ee987f64781b50ea89729e90a02 [id=c6c7f21718f645d5a26d3f8821ba470b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.848214 14274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9ba7e0592347ce80e47d461ad1a4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.849045 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9ba7e0592347ce80e47d461ad1a4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.849526 14009 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
I20251211 08:58:06.850108 14009 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:06.832967 16320 ts_tablet_manager.cc:1950] T c0e0ee99989d41c6af82e330ab8e353c P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:06.860687 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 610) took 2429 ms. Trace:
I20251211 08:58:06.860975 16320 rpcz_store.cc:276] 1211 08:58:04.430989 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.431078 (+ 89us) service_pool.cc:225] Handling call
1211 08:58:06.434751 (+2003673us) ts_tablet_manager.cc:1151] Deleting tablet c0e0ee99989d41c6af82e330ab8e353c
1211 08:58:06.455511 (+ 20760us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b5800000098. stack: 00007efeecdf4083 00007efeecdf3f80 00007efeec969574 0000000000508744 00000000005086d9 00007efeecc66781 00007efeecde8ac1 00007efeecde90f2 00007efef5c36419 00007efef4e8c108 00007efef4f529a9 00007efef4f52fa3 00007efef4f83f80 00007efef4f6a54d 00007efef4f6a4f9 00007efef4f6a489
1211 08:58:06.455546 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:06.511662 (+ 56116us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:06.694300 (+182638us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:06.860636 (+166336us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45659904,"tablet-delete.queue_time_us":2003230}
I20251211 08:58:06.863054 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 611) took 2275 ms. Trace:
I20251211 08:58:06.867036 14904 tablet_replica.cc:333] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:06.868115 14904 raft_consensus.cc:2243] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 2 LEADER]: Raft consensus shutting down.
I20251211 08:58:06.850589 14009 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:06.868907 14009 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 680c8f54e75a4291bb4d769d94e21552 (table default.Table_032686fd837d4292bd6c95749e67b5de [id=387069b529984870961fe2e545fbb221]). Expected version 2 got 0
I20251211 08:58:06.869021 14904 pending_rounds.cc:70] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: Trying to abort 1 pending ops.
I20251211 08:58:06.869251 14904 pending_rounds.cc:77] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7231256507470110720 op_type: NO_OP noop_request { }
I20251211 08:58:06.870213 14009 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet d7a8acb51a2948f4a018787bfaecb694 (table default.Table_9e007a9469d54d99b9471cb8e653b620 [id=de3a242b7e6d480ebf509e458ec002c7]). Expected version 1 got 0
I20251211 08:58:06.870337 14722 tablet_bootstrap.cc:492] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
W20251211 08:58:06.871096 14385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.146.1:36181, user_credentials={real_user=slave}} blocked reactor thread for 58408.3us
I20251211 08:58:06.875973 14722 tablet_bootstrap.cc:654] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.877360 14272 tablet_service.cc:1505] Processing CreateTablet for tablet 67cc46a2328f453389ec2f27c39a75bb (DEFAULT_TABLE table=default.Table_d8915549348b457baf42c8b9dfaa98fb [id=71cb9e4517394ba3adbab58462cc6a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.878360 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cc46a2328f453389ec2f27c39a75bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.876765 14904 raft_consensus.cc:2889] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:58:06.879022 14904 raft_consensus.cc:2272] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:06.887843 14573 tablet_service.cc:1505] Processing CreateTablet for tablet 67cc46a2328f453389ec2f27c39a75bb (DEFAULT_TABLE table=default.Table_d8915549348b457baf42c8b9dfaa98fb [id=71cb9e4517394ba3adbab58462cc6a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.889076 14573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cc46a2328f453389ec2f27c39a75bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.863358 16320 rpcz_store.cc:276] 1211 08:58:04.587419 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.587899 (+ 480us) service_pool.cc:225] Handling call
1211 08:58:06.862797 (+2274898us) ts_tablet_manager.cc:1151] Deleting tablet 033c2c016f8f4c01adc8bdbf2885e394
1211 08:58:06.862838 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:06.862997 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2149114}
I20251211 08:58:06.898366 14424 tablet_service.cc:1505] Processing CreateTablet for tablet a30d6f6f3224444880098beb77a14531 (DEFAULT_TABLE table=default.Table_7ed565fc5cb943fb85b6cf27e3c57e6c [id=5dae6ccab28e4a0ea61c7058328c96bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.899047 15030 tablet_bootstrap.cc:654] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.901383 14419 tablet_service.cc:1505] Processing CreateTablet for tablet 8b9ba7e0592347ce80e47d461ad1a4e2 (DEFAULT_TABLE table=default.Table_df875ee987f64781b50ea89729e90a02 [id=c6c7f21718f645d5a26d3f8821ba470b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.896494 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet c0e0ee99989d41c6af82e330ab8e353c (table default.Table_7ca1c0fc20ca4dd7a11793c0ac968666 [id=67ab831326db412fa228aac2683950ee]) successfully deleted
I20251211 08:58:06.903506 14009 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (table default.Table_ae8d82f4355240188c054f001b2c05a4 [id=f74e074787144a21ad31d4d89cc509c9]). Expected version 2 got 1
I20251211 08:58:06.904366 14714 tablet_bootstrap.cc:654] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:06.904690 14009 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:06.905485 14009 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 4dcd09f7c6364d74bb33503b5bbfea96 (table default.Table_78e567c0deac4f8d99c850bc335bbaf5 [id=482af2b30df84c8891e037e1aea436af]). Expected version 1 got 0
I20251211 08:58:06.907999 14413 tablet_service.cc:1505] Processing CreateTablet for tablet 0f1a55d787fc4bbdbe4858787d7df8c9 (DEFAULT_TABLE table=default.Table_184dbe4f30324cfcbb04f5031169242e [id=6dd6ac573e82475e82f7c9512616261c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.920475 14904 ts_tablet_manager.cc:1916] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:58:06.921504 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 033c2c016f8f4c01adc8bdbf2885e394 because the tablet was not found. No further retry: Not found: Tablet not found: 033c2c016f8f4c01adc8bdbf2885e394
I20251211 08:58:06.921976 14423 tablet_service.cc:1505] Processing CreateTablet for tablet 67cc46a2328f453389ec2f27c39a75bb (DEFAULT_TABLE table=default.Table_d8915549348b457baf42c8b9dfaa98fb [id=71cb9e4517394ba3adbab58462cc6a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:06.923439 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 612) took 2325 ms. Trace:
I20251211 08:58:06.923722 16320 rpcz_store.cc:276] 1211 08:58:04.598075 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.598898 (+ 823us) service_pool.cc:225] Handling call
1211 08:58:06.892378 (+2293480us) ts_tablet_manager.cc:1151] Deleting tablet ed0e90efe651499c929b1fb992712b96
1211 08:58:06.923206 (+ 30828us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:06.923380 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2178645}
W20251211 08:58:06.932592 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet ed0e90efe651499c929b1fb992712b96 because the tablet was not found. No further retry: Not found: Tablet not found: ed0e90efe651499c929b1fb992712b96
I20251211 08:58:06.937197 14424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30d6f6f3224444880098beb77a14531. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.943209 14419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9ba7e0592347ce80e47d461ad1a4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.948578 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f1a55d787fc4bbdbe4858787d7df8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.959327 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cc46a2328f453389ec2f27c39a75bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:06.959210 16320 tablet_replica.cc:333] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:06.969795 16320 raft_consensus.cc:2243] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:06.970278 16320 raft_consensus.cc:2272] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:06.973132 16320 ts_tablet_manager.cc:1916] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:06.984169 14714 tablet_bootstrap.cc:492] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:06.984426 14722 tablet_bootstrap.cc:492] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:06.984561 14714 ts_tablet_manager.cc:1403] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.150s user 0.008s sys 0.000s
I20251211 08:58:06.984784 14722 ts_tablet_manager.cc:1403] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.115s user 0.009s sys 0.001s
I20251211 08:58:06.990880 14722 raft_consensus.cc:359] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.991508 14722 raft_consensus.cc:385] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:06.991727 14722 raft_consensus.cc:740] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:06.992282 14722 consensus_queue.cc:260] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:06.994338 14722 ts_tablet_manager.cc:1434] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251211 08:58:07.003167 16305 consensus_queue.cc:1048] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:06.986820 14714 raft_consensus.cc:359] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:07.007349 14714 raft_consensus.cc:385] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:07.007629 14714 raft_consensus.cc:740] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:07.008390 14714 consensus_queue.cc:260] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:07.015555 14006 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:07.016237 15030 tablet_bootstrap.cc:492] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:07.016626 15030 ts_tablet_manager.cc:1403] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.228s user 0.006s sys 0.003s
I20251211 08:58:07.017571 16305 consensus_queue.cc:1048] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:07.018908 15030 raft_consensus.cc:359] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.019475 15030 raft_consensus.cc:385] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:07.019730 15030 raft_consensus.cc:740] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:07.020457 15030 consensus_queue.cc:260] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.021713 15030 ts_tablet_manager.cc:1434] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251211 08:58:07.022601 15030 tablet_bootstrap.cc:492] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:07.042659 14904 ts_tablet_manager.cc:1929] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251211 08:58:07.043226 14904 log.cc:1199] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/51c3f5eaef784e868e565f81bd51a5e4
I20251211 08:58:07.043897 14006 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
I20251211 08:58:07.044466 14904 ts_tablet_manager.cc:1950] T 51c3f5eaef784e868e565f81bd51a5e4 P 31404d5f58e548519239569b150d975b: Deleting consensus metadata
I20251211 08:58:07.045387 14006 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:07.051188 14298 raft_consensus.cc:3060] T 4dcd09f7c6364d74bb33503b5bbfea96 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:07.056962 14714 ts_tablet_manager.cc:1434] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.072s user 0.006s sys 0.000s
I20251211 08:58:07.060417 14722 tablet_bootstrap.cc:492] T 947e6203f910442ca01e2c21587cfe90 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:07.065802 14006 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:07.059149 16305 consensus_queue.cc:1048] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:07.072942 14006 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:07.097795 15030 tablet_bootstrap.cc:654] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:07.100131 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet cb2ff90b54544420b1aad9d2ab4132ff (table default.Table_cd6e057b26b443ae9e79158e9350dce5 [id=3c0379138b1c48e0a5400a465c4f372e]). Expected version 3 got 2
I20251211 08:58:07.101014 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:07.103255 14714 tablet_bootstrap.cc:492] T 4631dd0bc0c144e69a31de80de39a76a P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:07.104527 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet bfb3a2b8f8f64c12ab590ffc0be7ace3 (table default.Table_ae8d82f4355240188c054f001b2c05a4 [id=f74e074787144a21ad31d4d89cc509c9]). Expected version 2 got 1
I20251211 08:58:07.106316 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:07.108394 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 17ab48721a5c48a7b8a5550be658837d (table default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710]). Expected version 1 got 0
I20251211 08:58:07.108819 16436 raft_consensus.cc:493] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:07.109221 16436 raft_consensus.cc:515] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:07.110884 16436 leader_election.cc:290] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:07.111361 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet c7c04583bef74d84a49cdcd8a1305a88 (table default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120]). Expected version 1 got 0
I20251211 08:58:07.111976 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
I20251211 08:58:07.112987 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 111adfd1ac1a42bdafa9416594e7e932 (table default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b]). Expected version 3 got 0
I20251211 08:58:07.113107 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "145096302ca24d7e96f2fec45960b18a" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:07.115650 16412 consensus_queue.cc:1048] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:58:07.118640 14382 leader_election.cc:343] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:07.122642 14287 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "145096302ca24d7e96f2fec45960b18a" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:07.124637 14385 leader_election.cc:343] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:07.124969 14385 leader_election.cc:304] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:07.126224 16436 raft_consensus.cc:2749] T 145096302ca24d7e96f2fec45960b18a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:07.130060 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 566) took 3483 ms. Trace:
I20251211 08:58:07.131438 14011 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:07.131174 14904 rpcz_store.cc:276] 1211 08:58:03.647071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.647150 (+ 79us) service_pool.cc:225] Handling call
1211 08:58:06.839840 (+3192690us) ts_tablet_manager.cc:1151] Deleting tablet 51c3f5eaef784e868e565f81bd51a5e4
1211 08:58:06.866921 (+ 27081us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:06.928014 (+ 61093us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:07.042463 (+114449us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:07.130009 (+ 87546us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3192263}
I20251211 08:58:07.132236 14000 catalog_manager.cc:4985] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): tablet 51c3f5eaef784e868e565f81bd51a5e4 (table default.Table_7343d67071c54141af5a57615f82221e [id=fe5786801bab48888480739b8ff266e8]) successfully deleted
I20251211 08:58:07.132627 14011 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 4dcd09f7c6364d74bb33503b5bbfea96 (table default.Table_78e567c0deac4f8d99c850bc335bbaf5 [id=482af2b30df84c8891e037e1aea436af]). Expected version 1 got 0
I20251211 08:58:07.133261 14714 tablet_bootstrap.cc:654] T 4631dd0bc0c144e69a31de80de39a76a P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:07.137040 14722 tablet_bootstrap.cc:654] T 947e6203f910442ca01e2c21587cfe90 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:07.137909 14904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46844 (request call id 567) took 3490 ms. Trace:
I20251211 08:58:07.140616 14904 rpcz_store.cc:276] 1211 08:58:03.647854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.647936 (+ 82us) service_pool.cc:225] Handling call
1211 08:58:07.137635 (+3489699us) ts_tablet_manager.cc:1151] Deleting tablet da57b72207414512b9744b283bc869fe
1211 08:58:07.137688 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:07.137860 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3489277}
I20251211 08:58:07.146855 14904 tablet_replica.cc:333] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b: stopping tablet replica
I20251211 08:58:07.148806 14904 raft_consensus.cc:2243] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus shutting down.
W20251211 08:58:07.142251 14000 catalog_manager.cc:4952] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): delete failed for tablet da57b72207414512b9744b283bc869fe because the tablet was not found. No further retry: Not found: Tablet not found: da57b72207414512b9744b283bc869fe
I20251211 08:58:07.150844 14011 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet d7a8acb51a2948f4a018787bfaecb694 (table default.Table_9e007a9469d54d99b9471cb8e653b620 [id=de3a242b7e6d480ebf509e458ec002c7]). Expected version 1 got 0
I20251211 08:58:07.154686 14011 catalog_manager.cc:5654] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20251211 08:58:07.156697 14904 raft_consensus.cc:2272] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:07.157852 14011 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:07.159338 14011 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
I20251211 08:58:07.171259 14722 tablet_bootstrap.cc:492] T 947e6203f910442ca01e2c21587cfe90 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:07.176884 14722 ts_tablet_manager.cc:1403] T 947e6203f910442ca01e2c21587cfe90 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.117s user 0.010s sys 0.000s
I20251211 08:58:07.178220 16412 raft_consensus.cc:493] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:07.178687 16412 raft_consensus.cc:515] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.180456 16412 leader_election.cc:290] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:07.182670 14287 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7264e13dd44b499683e32b3fc62ed8ed" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:07.185379 14544 leader_election.cc:343] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:07.193035 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
I20251211 08:58:07.199139 16320 ts_tablet_manager.cc:1929] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:07.200589 14722 raft_consensus.cc:359] T 947e6203f910442ca01e2c21587cfe90 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.203352 16320 log.cc:1199] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/07f983604bfa474f9c9ce0c982d738fc
I20251211 08:58:07.203671 14722 raft_consensus.cc:385] T 947e6203f910442ca01e2c21587cfe90 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:07.205660 14722 raft_consensus.cc:740] T 947e6203f910442ca01e2c21587cfe90 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:07.206324 14722 consensus_queue.cc:260] T 947e6203f910442ca01e2c21587cfe90 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.208117 16437 raft_consensus.cc:493] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:07.210083 16437 raft_consensus.cc:515] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.206859 16320 ts_tablet_manager.cc:1950] T 07f983604bfa474f9c9ce0c982d738fc P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:07.212605 16437 leader_election.cc:290] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:07.226909 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "680c8f54e75a4291bb4d769d94e21552" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:07.227926 14230 leader_election.cc:343] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:07.229312 14441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "680c8f54e75a4291bb4d769d94e21552" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:07.229856 14441 raft_consensus.cc:2468] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 0.
I20251211 08:58:07.230722 14231 leader_election.cc:304] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters: c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:07.230850 14439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7264e13dd44b499683e32b3fc62ed8ed" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:07.231359 14439 raft_consensus.cc:2468] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:07.231797 16437 raft_consensus.cc:2804] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:07.232304 16437 raft_consensus.cc:493] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:07.232663 16437 raft_consensus.cc:3060] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:07.232765 14542 leader_election.cc:304] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:58:07.233930 16301 raft_consensus.cc:2804] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:07.234356 16301 raft_consensus.cc:493] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:07.234678 16301 raft_consensus.cc:3060] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:07.239501 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 680c8f54e75a4291bb4d769d94e21552 (table default.Table_032686fd837d4292bd6c95749e67b5de [id=387069b529984870961fe2e545fbb221]). Expected version 2 got 0
I20251211 08:58:07.240423 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet d7a8acb51a2948f4a018787bfaecb694 (table default.Table_9e007a9469d54d99b9471cb8e653b620 [id=de3a242b7e6d480ebf509e458ec002c7]). Expected version 1 got 0
I20251211 08:58:07.241358 16301 raft_consensus.cc:515] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.244616 14288 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7264e13dd44b499683e32b3fc62ed8ed" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b"
W20251211 08:58:07.245743 14544 leader_election.cc:343] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:07.251273 14722 ts_tablet_manager.cc:1434] T 947e6203f910442ca01e2c21587cfe90 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.072s user 0.006s sys 0.000s
I20251211 08:58:07.252640 14439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7264e13dd44b499683e32b3fc62ed8ed" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:07.253237 14439 raft_consensus.cc:3060] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:07.256839 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 614) took 2641 ms. Trace:
I20251211 08:58:07.257282 16301 leader_election.cc:290] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:07.259588 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:07.260226 14012 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 4dcd09f7c6364d74bb33503b5bbfea96 (table default.Table_78e567c0deac4f8d99c850bc335bbaf5 [id=482af2b30df84c8891e037e1aea436af]). Expected version 1 got 0
I20251211 08:58:07.261334 15030 tablet_bootstrap.cc:492] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:07.261668 15030 ts_tablet_manager.cc:1403] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.239s user 0.004s sys 0.004s
I20251211 08:58:07.261953 14722 tablet_bootstrap.cc:492] T 27cb4228204c4176b1a348960efd5b26 P 31404d5f58e548519239569b150d975b: Bootstrap starting.
I20251211 08:58:07.267019 14722 tablet_bootstrap.cc:654] T 27cb4228204c4176b1a348960efd5b26 P 31404d5f58e548519239569b150d975b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:07.264111 14714 tablet_bootstrap.cc:492] T 4631dd0bc0c144e69a31de80de39a76a P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:07.272107 14714 ts_tablet_manager.cc:1403] T 4631dd0bc0c144e69a31de80de39a76a P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.169s user 0.007s sys 0.003s
I20251211 08:58:07.274818 14904 ts_tablet_manager.cc:1916] T f2f095b59ecf44ffabaa0ac22212938b P 31404d5f58e548519239569b150d975b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:07.274497 14714 raft_consensus.cc:359] T 4631dd0bc0c144e69a31de80de39a76a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:07.275283 14714 raft_consensus.cc:385] T 4631dd0bc0c144e69a31de80de39a76a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:07.275576 14714 raft_consensus.cc:740] T 4631dd0bc0c144e69a31de80de39a76a P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:07.276279 14714 consensus_queue.cc:260] T 4631dd0bc0c144e69a31de80de39a76a P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:07.279382 16437 raft_consensus.cc:515] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.281173 16437 leader_election.cc:290] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Requested vote from peers c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:07.282212 14449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "680c8f54e75a4291bb4d769d94e21552" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:07.282678 14449 raft_consensus.cc:3060] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:07.288815 14449 raft_consensus.cc:2468] T 680c8f54e75a4291bb4d769d94e21552 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31404d5f58e548519239569b150d975b in term 1.
I20251211 08:58:07.289341 14714 ts_tablet_manager.cc:1434] T 4631dd0bc0c144e69a31de80de39a76a P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20251211 08:58:07.290007 14231 leader_election.cc:304] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256; no voters:
I20251211 08:58:07.290243 14714 tablet_bootstrap.cc:492] T f6fb2096f80a43b5a5a6230a824fceb7 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
I20251211 08:58:07.292799 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "680c8f54e75a4291bb4d769d94e21552" candidate_uuid: "31404d5f58e548519239569b150d975b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063"
W20251211 08:58:07.294479 14230 leader_election.cc:343] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:07.299507 16320 rpcz_store.cc:276] 1211 08:58:04.615479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.615572 (+ 93us) service_pool.cc:225] Handling call
1211 08:58:06.924434 (+2308862us) ts_tablet_manager.cc:1151] Deleting tablet 07f983604bfa474f9c9ce0c982d738fc
1211 08:58:06.959021 (+ 34587us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:07.050197 (+ 91176us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:07.198709 (+148512us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:07.256779 (+ 58070us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2210486}
I20251211 08:58:07.303812 16437 raft_consensus.cc:2804] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:07.304666 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:07.304400 16437 raft_consensus.cc:697] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [term 1 LEADER]: Becoming Leader. State: Replica: 31404d5f58e548519239569b150d975b, State: Running, Role: LEADER
I20251211 08:58:07.305234 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
I20251211 08:58:07.304632 16436 raft_consensus.cc:493] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:07.307523 16436 raft_consensus.cc:515] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:07.308790 15030 raft_consensus.cc:359] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.309300 16436 leader_election.cc:290] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
I20251211 08:58:07.309397 15030 raft_consensus.cc:385] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:07.309623 15030 raft_consensus.cc:740] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:07.306115 16437 consensus_queue.cc:237] T 680c8f54e75a4291bb4d769d94e21552 P 31404d5f58e548519239569b150d975b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.311401 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bb339322504704ae57bba2f4c3a9bb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
W20251211 08:58:07.312434 14382 leader_election.cc:343] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:07.312397 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 07f983604bfa474f9c9ce0c982d738fc (table default.Table_80758110ce794260af2a551849bf4896 [id=277e1a7e34f54c16b29a738c17f75da5]) successfully deleted
I20251211 08:58:07.311460 14298 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0bb339322504704ae57bba2f4c3a9bb" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31404d5f58e548519239569b150d975b" is_pre_election: true
W20251211 08:58:07.313782 14385 leader_election.cc:343] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:07.314198 14385 leader_election.cc:304] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b276f77f87254a5d909c5ab940a39256; no voters: 31404d5f58e548519239569b150d975b, c5e624bcfd154cd586ef1dedb9940063
I20251211 08:58:07.314913 16436 raft_consensus.cc:2749] T c0bb339322504704ae57bba2f4c3a9bb P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:07.316002 16320 tablet_replica.cc:333] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:07.317726 16320 raft_consensus.cc:2243] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:07.320063 16320 pending_rounds.cc:70] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: Trying to abort 1 pending ops.
I20251211 08:58:07.327091 16320 pending_rounds.cc:77] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256521994854400 op_type: NO_OP noop_request { }
I20251211 08:58:07.332000 16320 raft_consensus.cc:2889] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:58:07.332852 16320 raft_consensus.cc:2272] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:07.336828 16320 ts_tablet_manager.cc:1916] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:07.310185 15030 consensus_queue.cc:260] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.306070 14013 catalog_manager.cc:5654] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31404d5f58e548519239569b150d975b (127.13.146.1). New cstate: current_term: 1 leader_uuid: "31404d5f58e548519239569b150d975b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20251211 08:58:07.342732 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet d7a8acb51a2948f4a018787bfaecb694 (table default.Table_9e007a9469d54d99b9471cb8e653b620 [id=de3a242b7e6d480ebf509e458ec002c7]). Expected version 1 got 0
I20251211 08:58:07.343782 14013 catalog_manager.cc:5746] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:07.349629 15030 ts_tablet_manager.cc:1434] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.088s user 0.002s sys 0.004s
I20251211 08:58:07.355748 15030 tablet_bootstrap.cc:492] T 680c8f54e75a4291bb4d769d94e21552 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:07.334692 14439 raft_consensus.cc:2468] T 7264e13dd44b499683e32b3fc62ed8ed P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:07.356734 14542 leader_election.cc:304] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b
I20251211 08:58:07.357077 16392 consensus_queue.cc:1048] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:07.357620 16301 raft_consensus.cc:2804] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:07.358306 16301 raft_consensus.cc:697] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:07.359123 16301 consensus_queue.cc:237] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.363659 14722 tablet_bootstrap.cc:492] T 27cb4228204c4176b1a348960efd5b26 P 31404d5f58e548519239569b150d975b: No bootstrap required, opened a new log
I20251211 08:58:07.366321 14722 ts_tablet_manager.cc:1403] T 27cb4228204c4176b1a348960efd5b26 P 31404d5f58e548519239569b150d975b: Time spent bootstrapping tablet: real 0.105s user 0.005s sys 0.008s
I20251211 08:58:07.370970 14007 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_8d7c119ecb1e4470b2015b8a77780805 [id=03d8c8d3f3844ed19a97971ed884317b] alter complete (version 3)
I20251211 08:58:07.372432 14714 tablet_bootstrap.cc:654] T f6fb2096f80a43b5a5a6230a824fceb7 P b276f77f87254a5d909c5ab940a39256: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:07.369856 14722 raft_consensus.cc:359] T 27cb4228204c4176b1a348960efd5b26 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:07.382995 14722 raft_consensus.cc:385] T 27cb4228204c4176b1a348960efd5b26 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:07.388031 14722 raft_consensus.cc:740] T 27cb4228204c4176b1a348960efd5b26 P 31404d5f58e548519239569b150d975b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31404d5f58e548519239569b150d975b, State: Initialized, Role: FOLLOWER
I20251211 08:58:07.388864 14722 consensus_queue.cc:260] T 27cb4228204c4176b1a348960efd5b26 P 31404d5f58e548519239569b150d975b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:07.391122 14722 ts_tablet_manager.cc:1434] T 27cb4228204c4176b1a348960efd5b26 P 31404d5f58e548519239569b150d975b: Time spent starting tablet: real 0.023s user 0.002s sys 0.003s
I20251211 08:58:07.394500 14012 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_5f90244381bf4b53ab7d861a768e6cd9 [id=202795cefb8a49dfa9c9353f60bc4120] alter complete (version 1)
I20251211 08:58:07.400126 14011 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_cd6e057b26b443ae9e79158e9350dce5 [id=3c0379138b1c48e0a5400a465c4f372e] alter complete (version 3)
I20251211 08:58:07.401633 16438 consensus_queue.cc:1048] T d7a8acb51a2948f4a018787bfaecb694 P 31404d5f58e548519239569b150d975b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:07.401949 14010 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55614 (request call id 662) took 3958 ms. Trace:
I20251211 08:58:07.402230 14010 rpcz_store.cc:276] 1211 08:58:03.443468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.435672 (+992204us) service_pool.cc:225] Handling call
1211 08:58:04.523078 (+ 87406us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da87016 00007fc20b6b5b24 00007fc20b6b5ab2 00007fc20b6b59e1 00007fc20b6b595c 00007fc20b6b4c12 00007fc20998455c 00007fc209983996
1211 08:58:04.530436 (+ 7358us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1211 08:58:04.531030 (+ 594us) catalog_manager.cc:3708] Apply alter schema
1211 08:58:04.531583 (+ 553us) catalog_manager.cc:3740] Acquired catalog manager lock
1211 08:58:04.531777 (+ 194us) catalog_manager.cc:3898] Updating metadata on disk
1211 08:58:04.683063 (+151286us) catalog_manager.cc:3929] Committing alterations to in-memory state
1211 08:58:07.401891 (+2718828us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:04.625113 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:04.627963 (+ 2850us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:04.627985 (+ 22us) write_op.cc:435] Acquired schema lock
1211 08:58:04.628013 (+ 28us) tablet.cc:660] Decoding operations
1211 08:58:04.631296 (+ 3283us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:04.631341 (+ 45us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:04.631365 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:04.631507 (+ 142us) tablet.cc:699] Row locks acquired
1211 08:58:04.631520 (+ 13us) write_op.cc:260] PREPARE: finished
1211 08:58:04.631631 (+ 111us) write_op.cc:270] Start()
1211 08:58:04.632948 (+ 1317us) write_op.cc:276] Timestamp: P: 1765443484631605 usec, L: 0
1211 08:58:04.632965 (+ 17us) op_driver.cc:348] REPLICATION: starting
1211 08:58:04.633388 (+ 423us) log.cc:844] Serialized 405 byte log entry
1211 08:58:04.680575 (+ 47187us) op_driver.cc:464] REPLICATION: finished
1211 08:58:04.680795 (+ 220us) write_op.cc:301] APPLY: starting
1211 08:58:04.680851 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:04.680939 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:04.681053 (+ 114us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:04.681214 (+ 161us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:04.681381 (+ 167us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:04.681429 (+ 48us) write_op.cc:312] APPLY: finished
1211 08:58:04.681608 (+ 179us) log.cc:844] Serialized 23 byte log entry
1211 08:58:04.681745 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:04.681805 (+ 60us) write_op.cc:454] Released schema lock
1211 08:58:04.681840 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":30732032,"child_traces":[["op",{"apply.queue_time_us":84,"apply.run_cpu_time_us":1221,"apply.run_wall_time_us":1221,"num_ops":1,"prepare.queue_time_us":92380,"prepare.run_cpu_time_us":1819,"prepare.run_wall_time_us":8455,"replication_time_us":47508}]]}
I20251211 08:58:07.414732 13896 master-stress-test.cc:579] Killed test thread 5/10
I20251211 08:58:07.415122 13896 master-stress-test.cc:577] Killing test thread 6/10
I20251211 08:58:07.415437 13896 master-stress-test.cc:579] Killed test thread 6/10
I20251211 08:58:07.415649 13896 master-stress-test.cc:577] Killing test thread 7/10
I20251211 08:58:07.416175 13896 master-stress-test.cc:579] Killed test thread 7/10
I20251211 08:58:07.416389 13896 master-stress-test.cc:577] Killing test thread 8/10
I20251211 08:58:07.416687 13896 master-stress-test.cc:579] Killed test thread 8/10
I20251211 08:58:07.416890 13896 master-stress-test.cc:577] Killing test thread 9/10
I20251211 08:58:07.417217 13896 master-stress-test.cc:579] Killed test thread 9/10
I20251211 08:58:07.417430 13896 master-stress-test.cc:577] Killing test thread 10/10
I20251211 08:58:07.417761 13896 master-stress-test.cc:579] Killed test thread 10/10
I20251211 08:58:07.418172 13896 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu with pid 14176
I20251211 08:58:07.421808 14653 heartbeater.cc:344] Connected to a master server at 127.13.146.61:34977
I20251211 08:58:07.437161 16320 ts_tablet_manager.cc:1929] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:07.437605 16320 log.cc:1199] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/7af7c9ab27a64a1681f9874a922e986f
I20251211 08:58:07.446282 16320 ts_tablet_manager.cc:1950] T 7af7c9ab27a64a1681f9874a922e986f P b276f77f87254a5d909c5ab940a39256: Deleting consensus metadata
I20251211 08:58:07.452481 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 616) took 2746 ms. Trace:
I20251211 08:58:07.457260 16320 rpcz_store.cc:276] 1211 08:58:04.705769 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.705904 (+ 135us) service_pool.cc:225] Handling call
1211 08:58:07.315857 (+2609953us) ts_tablet_manager.cc:1151] Deleting tablet 7af7c9ab27a64a1681f9874a922e986f
1211 08:58:07.315902 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:07.394567 (+ 78665us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:07.436932 (+ 42365us) tablet_metadata.cc:641] Metadata flushed
1211 08:58:07.452414 (+ 15482us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2497444}
I20251211 08:58:07.460036 13998 catalog_manager.cc:4985] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): tablet 7af7c9ab27a64a1681f9874a922e986f (table default.Table_d9c4c3852105418a914f5055b9b70072 [id=60f08078e7f849c09040202347df4f18]) successfully deleted
I20251211 08:58:07.461112 16320 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 617) took 1854 ms. Trace:
I20251211 08:58:07.462715 16320 rpcz_store.cc:276] 1211 08:58:05.606400 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:05.606512 (+ 112us) service_pool.cc:225] Handling call
1211 08:58:07.460834 (+1854322us) ts_tablet_manager.cc:1151] Deleting tablet 53aac024d68c43bca999eec5bd2def69
1211 08:58:07.460868 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1211 08:58:07.461071 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1853783}
W20251211 08:58:07.464993 13998 catalog_manager.cc:4965] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 53aac024d68c43bca999eec5bd2def69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53aac024d68c43bca999eec5bd2def69 already in progress: creating tablet
I20251211 08:58:07.484105 14007 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_ae8d82f4355240188c054f001b2c05a4 [id=f74e074787144a21ad31d4d89cc509c9] alter complete (version 2)
I20251211 08:58:07.506546 16301 raft_consensus.cc:493] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:07.507017 16301 raft_consensus.cc:515] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.508965 16301 leader_election.cc:290] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:07.512470 14439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "947e6203f910442ca01e2c21587cfe90" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256" is_pre_election: true
I20251211 08:58:07.513049 14439 raft_consensus.cc:2468] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 0.
I20251211 08:58:07.515305 14542 leader_election.cc:304] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:07.521862 16301 raft_consensus.cc:2804] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:07.522228 16301 raft_consensus.cc:493] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:07.522578 16301 raft_consensus.cc:3060] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:07.530843 14009 catalog_manager.cc:6199] T 00000000000000000000000000000000 P a4c9cd45be3947d09423ed4508ca2e91: default.Table_6cdb098ed94b446daec0ae76c262dbb8 [id=1b741ee1e2bd404f982e2c2aea8da710] alter complete (version 1)
I20251211 08:58:07.530853 16301 raft_consensus.cc:515] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.536644 15030 tablet_bootstrap.cc:654] T 680c8f54e75a4291bb4d769d94e21552 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:07.537463 14439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "947e6203f910442ca01e2c21587cfe90" candidate_uuid: "c5e624bcfd154cd586ef1dedb9940063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276f77f87254a5d909c5ab940a39256"
I20251211 08:58:07.538002 14439 raft_consensus.cc:3060] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:07.545703 14439 raft_consensus.cc:2468] T 947e6203f910442ca01e2c21587cfe90 P b276f77f87254a5d909c5ab940a39256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e624bcfd154cd586ef1dedb9940063 in term 1.
I20251211 08:58:07.551028 14542 leader_election.cc:304] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276f77f87254a5d909c5ab940a39256, c5e624bcfd154cd586ef1dedb9940063; no voters:
I20251211 08:58:07.551679 16339 raft_consensus.cc:2804] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:07.552060 16339 raft_consensus.cc:697] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Becoming Leader. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Running, Role: LEADER
I20251211 08:58:07.552783 16339 consensus_queue.cc:237] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.556434 14714 tablet_bootstrap.cc:492] T f6fb2096f80a43b5a5a6230a824fceb7 P b276f77f87254a5d909c5ab940a39256: No bootstrap required, opened a new log
I20251211 08:58:07.556830 14714 ts_tablet_manager.cc:1403] T f6fb2096f80a43b5a5a6230a824fceb7 P b276f77f87254a5d909c5ab940a39256: Time spent bootstrapping tablet: real 0.267s user 0.040s sys 0.020s
I20251211 08:58:07.566111 14573 tablet_service.cc:1552] Processing DeleteTablet for tablet 07f983604bfa474f9c9ce0c982d738fc with delete_type TABLET_DATA_DELETED (replaced by tablet 960bc7a27fa04f2c9f398a67d2c93266) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:07.567414 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 with delete_type TABLET_DATA_DELETED (replaced by tablet e62ae5b540aa4680b4d18fee29c39e64) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:07.569160 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet ac6be204692a44fab2db551e79237299 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c0f8d7dd90f4beaae674fe827d37e0b) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:07.570469 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 476eee1606e74832abf928dbaafc844d with delete_type TABLET_DATA_DELETED (replaced by tablet 62c8dcb36e694cf896bd1283db63b4cc) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:07.572155 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c02cb6ec01041dba118683a57aecde2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:40 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:58:07.573643 14011 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 71) took 1744 ms (client timeout 1000 ms). Trace:
I20251211 08:58:07.594738 16301 leader_election.cc:290] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: Requested vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
I20251211 08:58:07.594978 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet c112dfcc301648cda6f0f038b1f2af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:07.596367 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 51c3f5eaef784e868e565f81bd51a5e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:47 UTC) from {username='slave'} at 127.0.0.1:42846
W20251211 08:58:07.618700 14006 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 102) took 2537 ms (client timeout 1000 ms). Trace:
W20251211 08:58:07.619139 14006 rpcz_store.cc:269] 1211 08:58:05.080967 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:05.081101 (+ 134us) service_pool.cc:225] Handling call
1211 08:58:07.618640 (+2537539us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68913,"reported_tablets":29}
I20251211 08:58:07.627139 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 033c2c016f8f4c01adc8bdbf2885e394 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f0bb5daf5c04698bad02fb5fa2e987e) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:07.629235 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet ac6be204692a44fab2db551e79237299 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c0f8d7dd90f4beaae674fe827d37e0b) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:07.630538 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet ed0e90efe651499c929b1fb992712b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:07.633072 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 6874fb018bc844b3aeeab7db8fc53e9a with delete_type TABLET_DATA_DELETED (replaced by tablet 593a90d88b194bb5a1c37e0e7e9097ef) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:07.633409 14576 tablet_service.cc:1552] Processing DeleteTablet for tablet ed0e90efe651499c929b1fb992712b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:43 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:07.633574 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 6874fb018bc844b3aeeab7db8fc53e9a with delete_type TABLET_DATA_DELETED (replaced by tablet 593a90d88b194bb5a1c37e0e7e9097ef) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:07.640882 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 7af7c9ab27a64a1681f9874a922e986f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:07.642118 16491 tablet_replica.cc:333] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:07.643170 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 with delete_type TABLET_DATA_DELETED (replaced by tablet e62ae5b540aa4680b4d18fee29c39e64) from {username='slave'} at 127.0.0.1:40738
W20251211 08:58:07.644963 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 033c2c016f8f4c01adc8bdbf2885e394 because the tablet was not found. No further retry: Not found: Tablet not found: 033c2c016f8f4c01adc8bdbf2885e394
W20251211 08:58:07.645532 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet ac6be204692a44fab2db551e79237299 because the tablet was not found. No further retry: Not found: Tablet not found: ac6be204692a44fab2db551e79237299
W20251211 08:58:07.645990 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet ed0e90efe651499c929b1fb992712b96 because the tablet was not found. No further retry: Not found: Tablet not found: ed0e90efe651499c929b1fb992712b96
I20251211 08:58:07.645359 16491 raft_consensus.cc:2243] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
W20251211 08:58:07.646414 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 6874fb018bc844b3aeeab7db8fc53e9a because the tablet was not found. No further retry: Not found: Tablet not found: 6874fb018bc844b3aeeab7db8fc53e9a
W20251211 08:58:07.646888 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 7af7c9ab27a64a1681f9874a922e986f because the tablet was not found. No further retry: Not found: Tablet not found: 7af7c9ab27a64a1681f9874a922e986f
I20251211 08:58:07.647066 16491 raft_consensus.cc:2272] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251211 08:58:07.647363 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 because the tablet was not found. No further retry: Not found: Tablet not found: 5d8d84e594ea47f3bc9127cc587e8aa7
I20251211 08:58:07.649997 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 033c2c016f8f4c01adc8bdbf2885e394 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f0bb5daf5c04698bad02fb5fa2e987e) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:07.650336 14714 raft_consensus.cc:359] T f6fb2096f80a43b5a5a6230a824fceb7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:07.654872 14714 raft_consensus.cc:385] T f6fb2096f80a43b5a5a6230a824fceb7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:07.655179 14714 raft_consensus.cc:740] T f6fb2096f80a43b5a5a6230a824fceb7 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276f77f87254a5d909c5ab940a39256, State: Initialized, Role: FOLLOWER
I20251211 08:58:07.658070 16491 ts_tablet_manager.cc:1916] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:07.655853 14714 consensus_queue.cc:260] T f6fb2096f80a43b5a5a6230a824fceb7 P b276f77f87254a5d909c5ab940a39256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:07.676450 14714 ts_tablet_manager.cc:1434] T f6fb2096f80a43b5a5a6230a824fceb7 P b276f77f87254a5d909c5ab940a39256: Time spent starting tablet: real 0.119s user 0.006s sys 0.000s
W20251211 08:58:07.574471 14011 rpcz_store.cc:269] 1211 08:58:05.829489 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:05.832089 (+ 2600us) service_pool.cc:225] Handling call
1211 08:58:07.180592 (+1348503us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.180627 (+ 35us) write_op.cc:454] Released schema lock
1211 08:58:07.216827 (+ 36200us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.216850 (+ 23us) write_op.cc:454] Released schema lock
1211 08:58:07.573584 (+356734us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:07.165317 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:07.165735 (+ 418us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:07.165760 (+ 25us) write_op.cc:435] Acquired schema lock
1211 08:58:07.165774 (+ 14us) tablet.cc:660] Decoding operations
1211 08:58:07.166052 (+ 278us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:07.166108 (+ 56us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:07.166130 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:07.166272 (+ 142us) tablet.cc:699] Row locks acquired
1211 08:58:07.166286 (+ 14us) write_op.cc:260] PREPARE: finished
1211 08:58:07.166400 (+ 114us) write_op.cc:270] Start()
1211 08:58:07.166480 (+ 80us) write_op.cc:276] Timestamp: P: 1765443487166559 usec, L: 0
1211 08:58:07.166493 (+ 13us) op_driver.cc:348] REPLICATION: starting
1211 08:58:07.167003 (+ 510us) log.cc:844] Serialized 530 byte log entry
1211 08:58:07.174879 (+ 7876us) op_driver.cc:464] REPLICATION: finished
1211 08:58:07.175203 (+ 324us) write_op.cc:301] APPLY: starting
1211 08:58:07.175273 (+ 70us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:07.175441 (+ 168us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:07.175569 (+ 128us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:07.175753 (+ 184us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:07.175907 (+ 154us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:07.175956 (+ 49us) write_op.cc:312] APPLY: finished
1211 08:58:07.176127 (+ 171us) log.cc:844] Serialized 23 byte log entry
1211 08:58:07.176256 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.176326 (+ 70us) write_op.cc:454] Released schema lock
1211 08:58:07.176356 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1211 08:58:07.196235 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:07.196659 (+ 424us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:07.196685 (+ 26us) write_op.cc:435] Acquired schema lock
1211 08:58:07.196703 (+ 18us) tablet.cc:660] Decoding operations
1211 08:58:07.196987 (+ 284us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:07.197027 (+ 40us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:07.197047 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:07.197207 (+ 160us) tablet.cc:699] Row locks acquired
1211 08:58:07.197218 (+ 11us) write_op.cc:260] PREPARE: finished
1211 08:58:07.197355 (+ 137us) write_op.cc:270] Start()
1211 08:58:07.197452 (+ 97us) write_op.cc:276] Timestamp: P: 1765443487197306 usec, L: 0
1211 08:58:07.200041 (+ 2589us) op_driver.cc:348] REPLICATION: starting
1211 08:58:07.200538 (+ 497us) log.cc:844] Serialized 356 byte log entry
1211 08:58:07.210826 (+ 10288us) op_driver.cc:464] REPLICATION: finished
1211 08:58:07.211117 (+ 291us) write_op.cc:301] APPLY: starting
1211 08:58:07.211177 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:07.211247 (+ 70us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:07.211338 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:07.211463 (+ 125us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:07.211561 (+ 98us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:07.211592 (+ 31us) write_op.cc:312] APPLY: finished
1211 08:58:07.211720 (+ 128us) log.cc:844] Serialized 23 byte log entry
1211 08:58:07.211857 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.211915 (+ 58us) write_op.cc:454] Released schema lock
1211 08:58:07.211941 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":26102,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":194,"apply.run_cpu_time_us":1305,"apply.run_wall_time_us":1303,"num_ops":1,"prepare.queue_time_us":3277,"prepare.run_cpu_time_us":2017,"prepare.run_wall_time_us":2016,"raft.queue_time_us":173,"raft.run_cpu_time_us":672,"raft.run_wall_time_us":671,"replication_time_us":8295}],["op",{"apply.queue_time_us":165,"apply.run_cpu_time_us":969,"apply.run_wall_time_us":968,"num_ops":1,"prepare.queue_time_us":1111,"prepare.run_cpu_time_us":2041,"prepare.run_wall_time_us":4608,"raft.queue_time_us":165,"raft.run_cpu_time_us":590,"raft.run_wall_time_us":590,"replication_time_us":10677}]]}
W20251211 08:58:07.680452 14005 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.3:43327 (request call id 70) took 3038 ms (client timeout 1000 ms). Trace:
W20251211 08:58:07.680749 14005 rpcz_store.cc:269] 1211 08:58:04.642454 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.642600 (+ 146us) service_pool.cc:225] Handling call
1211 08:58:07.680392 (+3037792us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26886,"reported_tablets":28}
W20251211 08:58:07.685093 14013 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 104) took 982 ms (client timeout 1000 ms). Trace:
I20251211 08:58:07.700289 14586 tablet_service.cc:1552] Processing DeleteTablet for tablet 570c7f08c46e4273bd38c32593dcf304 with delete_type TABLET_DATA_DELETED (replaced by tablet 3838fd220d034b3595f351b906188339) from {username='slave'} at 127.0.0.1:42846
W20251211 08:58:07.690843 14008 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 101) took 3857 ms (client timeout 1000 ms). Trace:
W20251211 08:58:07.708324 14008 rpcz_store.cc:269] 1211 08:58:03.833593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.194044 (+360451us) service_pool.cc:225] Handling call
1211 08:58:07.690759 (+3496715us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64468,"reported_tablets":30}
W20251211 08:58:07.706102 14013 rpcz_store.cc:269] 1211 08:58:06.702226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:06.702345 (+ 119us) service_pool.cc:225] Handling call
1211 08:58:07.685021 (+982676us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:07.585762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:07.586128 (+ 366us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:07.586153 (+ 25us) write_op.cc:435] Acquired schema lock
1211 08:58:07.586163 (+ 10us) tablet.cc:660] Decoding operations
1211 08:58:07.586392 (+ 229us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:07.586435 (+ 43us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:07.586457 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:07.586602 (+ 145us) tablet.cc:699] Row locks acquired
1211 08:58:07.586614 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:58:07.586721 (+ 107us) write_op.cc:270] Start()
1211 08:58:07.586818 (+ 97us) write_op.cc:276] Timestamp: P: 1765443487586851 usec, L: 0
1211 08:58:07.586830 (+ 12us) op_driver.cc:348] REPLICATION: starting
1211 08:58:07.587286 (+ 456us) log.cc:844] Serialized 530 byte log entry
1211 08:58:07.599224 (+ 11938us) op_driver.cc:464] REPLICATION: finished
1211 08:58:07.599503 (+ 279us) write_op.cc:301] APPLY: starting
1211 08:58:07.599570 (+ 67us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:07.599651 (+ 81us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:07.599749 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:07.599933 (+ 184us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:07.600050 (+ 117us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:07.600104 (+ 54us) write_op.cc:312] APPLY: finished
1211 08:58:07.600252 (+ 148us) log.cc:844] Serialized 23 byte log entry
1211 08:58:07.600388 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.600446 (+ 58us) write_op.cc:454] Released schema lock
1211 08:58:07.600473 (+ 27us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:07.600675 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.600695 (+ 20us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":82049,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":160,"apply.run_cpu_time_us":1129,"apply.run_wall_time_us":1128,"num_ops":1,"prepare.queue_time_us":81,"prepare.run_cpu_time_us":1864,"prepare.run_wall_time_us":1863,"raft.queue_time_us":223,"raft.run_cpu_time_us":575,"raft.run_wall_time_us":574,"replication_time_us":12308}]]}
I20251211 08:58:07.710697 15030 tablet_bootstrap.cc:492] T 680c8f54e75a4291bb4d769d94e21552 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:07.712963 15030 ts_tablet_manager.cc:1403] T 680c8f54e75a4291bb4d769d94e21552 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.357s user 0.010s sys 0.000s
I20251211 08:58:07.716487 15030 raft_consensus.cc:359] T 680c8f54e75a4291bb4d769d94e21552 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.717113 15030 raft_consensus.cc:385] T 680c8f54e75a4291bb4d769d94e21552 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:07.717401 15030 raft_consensus.cc:740] T 680c8f54e75a4291bb4d769d94e21552 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:07.725222 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet ac6be204692a44fab2db551e79237299 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c0f8d7dd90f4beaae674fe827d37e0b) from {username='slave'} at 127.0.0.1:40738
W20251211 08:58:07.731822 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet ac6be204692a44fab2db551e79237299 because the tablet was not found. No further retry: Not found: Tablet not found: ac6be204692a44fab2db551e79237299
I20251211 08:58:07.732757 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 with delete_type TABLET_DATA_DELETED (replaced by tablet e62ae5b540aa4680b4d18fee29c39e64) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:07.733353 14419 tablet_service.cc:1552] Processing DeleteTablet for tablet 476eee1606e74832abf928dbaafc844d with delete_type TABLET_DATA_DELETED (replaced by tablet 62c8dcb36e694cf896bd1283db63b4cc) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:07.734722 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b2445f66be54d3fbdc3be964b9a9132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:56 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:58:07.735790 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 because the tablet was not found. No further retry: Not found: Tablet not found: 5d8d84e594ea47f3bc9127cc587e8aa7
W20251211 08:58:07.736377 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 476eee1606e74832abf928dbaafc844d because the tablet was not found. No further retry: Not found: Tablet not found: 476eee1606e74832abf928dbaafc844d
I20251211 08:58:07.736860 14419 tablet_service.cc:1552] Processing DeleteTablet for tablet 07f983604bfa474f9c9ce0c982d738fc with delete_type TABLET_DATA_DELETED (replaced by tablet 960bc7a27fa04f2c9f398a67d2c93266) from {username='slave'} at 127.0.0.1:40738
W20251211 08:58:07.736889 13998 catalog_manager.cc:4952] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): delete failed for tablet 1b2445f66be54d3fbdc3be964b9a9132 because the tablet was not found. No further retry: Not found: Tablet not found: 1b2445f66be54d3fbdc3be964b9a9132
I20251211 08:58:07.737325 14413 tablet_service.cc:1552] Processing DeleteTablet for tablet 143f20f5c37948068ff2fb0d92f0f34b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:54 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:58:07.739209 14007 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 67) took 2878 ms (client timeout 1000 ms). Trace:
I20251211 08:58:07.739889 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 6874fb018bc844b3aeeab7db8fc53e9a with delete_type TABLET_DATA_DELETED (replaced by tablet 593a90d88b194bb5a1c37e0e7e9097ef) from {username='slave'} at 127.0.0.1:40738
W20251211 08:58:07.741727 14012 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 68) took 1786 ms (client timeout 1000 ms). Trace:
I20251211 08:58:07.743153 14427 tablet_service.cc:1552] Processing DeleteTablet for tablet 7af7c9ab27a64a1681f9874a922e986f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:07.743889 14424 tablet_service.cc:1552] Processing DeleteTablet for tablet 476eee1606e74832abf928dbaafc844d with delete_type TABLET_DATA_DELETED (replaced by tablet 62c8dcb36e694cf896bd1283db63b4cc) from {username='slave'} at 127.0.0.1:40738
I20251211 08:58:07.744038 14714 tablet_bootstrap.cc:492] T e90daf0f780a467e9c900b91c1615cb2 P b276f77f87254a5d909c5ab940a39256: Bootstrap starting.
W20251211 08:58:07.744691 14004 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.1:34545 (request call id 100) took 5129 ms (client timeout 1000 ms). Trace:
I20251211 08:58:07.744882 14414 tablet_service.cc:1552] Processing DeleteTablet for tablet f56fd93aeac6435ea84388ce05580113 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:58:07.745034 14004 rpcz_store.cc:269] 1211 08:58:02.615481 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.161843 (+546362us) service_pool.cc:225] Handling call
1211 08:58:07.744623 (+4582780us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":94238,"reported_tablets":29}
I20251211 08:58:07.744252 16489 raft_consensus.cc:493] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251211 08:58:07.750272 14384 connection.cc:537] server connection from 127.13.146.1:50379 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251211 08:58:07.750200 16489 raft_consensus.cc:515] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
W20251211 08:58:07.751397 14009 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 66) took 3981 ms (client timeout 1000 ms). Trace:
I20251211 08:58:07.751709 16489 leader_election.cc:290] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291)
W20251211 08:58:07.752983 14000 connection.cc:537] client connection to 127.13.146.1:36181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:07.753326 14000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:07.753446 14544 connection.cc:537] client connection to 127.13.146.1:36181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:07.753870 14544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251211 08:58:07.755101 14603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d69451d9e49740bb9af84d39d157c2f0" candidate_uuid: "b276f77f87254a5d909c5ab940a39256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e624bcfd154cd586ef1dedb9940063" is_pre_election: true
I20251211 08:58:07.757055 14414 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c95565ce3954cc598d7cbd45321d7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:44 UTC) from {username='slave'} at 127.0.0.1:40738
W20251211 08:58:07.757967 14385 connection.cc:700] client connection to 127.13.146.1:36181 send error: Network error: sendmsg error: Connection reset by peer (error 104)
W20251211 08:58:07.769106 14382 leader_election.cc:343] T d69451d9e49740bb9af84d39d157c2f0 P b276f77f87254a5d909c5ab940a39256 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:07.747515 16320 tablet_replica.cc:333] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256: stopping tablet replica
I20251211 08:58:07.771324 13896 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu with pid 14348
I20251211 08:58:07.771306 16320 raft_consensus.cc:2243] T 143f20f5c37948068ff2fb0d92f0f34b P b276f77f87254a5d909c5ab940a39256 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:07.772604 15030 consensus_queue.cc:260] T 680c8f54e75a4291bb4d769d94e21552 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.776029 15030 ts_tablet_manager.cc:1434] T 680c8f54e75a4291bb4d769d94e21552 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.063s user 0.004s sys 0.000s
W20251211 08:58:07.751710 14009 rpcz_store.cc:269] 1211 08:58:03.770242 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:03.835689 (+ 65447us) service_pool.cc:225] Handling call
1211 08:58:04.187326 (+351637us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54000203d4. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20bbf00c0 00007fc20bc8ebba 00007fc20d98b13c 00007fc20da8544e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:58:04.197170 (+ 9844us) spinlock_profiling.cc:243] Waited 5.64 ms on lock 0x7b5400001a90. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc204a1a781 00007fc204b9d590 00007fc20d97fd66 00007fc20da85d0e 00007fc20b6a54a4 00007fc20b6a5432 00007fc20b6a5361 00007fc20b6a52dc 00007fc20b6a4592 00007fc20998455c 00007fc209983996
1211 08:58:07.525183 (+3328013us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.525214 (+ 31us) write_op.cc:454] Released schema lock
1211 08:58:07.748058 (+222844us) spinlock_profiling.cc:243] Waited 49.4 ms on lock 0x7b4c00002680. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20991d060 00007fc20992deb2 00007fc20992e412 00007fc2098f2e7b 00007fc20992057c 00007fc209920a3e 00007fc20b286c1b 00007fc20d9d3fc1 00007fc20d97c140 00007fc20d98221b 00007fc20da85d0e
1211 08:58:07.751350 (+ 3292us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:07.507048 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:07.507658 (+ 610us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:07.507694 (+ 36us) write_op.cc:435] Acquired schema lock
1211 08:58:07.507717 (+ 23us) tablet.cc:660] Decoding operations
1211 08:58:07.508042 (+ 325us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:07.508087 (+ 45us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:07.508121 (+ 34us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:07.508309 (+ 188us) tablet.cc:699] Row locks acquired
1211 08:58:07.508331 (+ 22us) write_op.cc:260] PREPARE: finished
1211 08:58:07.508471 (+ 140us) write_op.cc:270] Start()
1211 08:58:07.508589 (+ 118us) write_op.cc:276] Timestamp: P: 1765443487508562 usec, L: 0
1211 08:58:07.508608 (+ 19us) op_driver.cc:348] REPLICATION: starting
1211 08:58:07.509275 (+ 667us) log.cc:844] Serialized 356 byte log entry
1211 08:58:07.518363 (+ 9088us) op_driver.cc:464] REPLICATION: finished
1211 08:58:07.518695 (+ 332us) write_op.cc:301] APPLY: starting
1211 08:58:07.518755 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:07.518885 (+ 130us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:07.518986 (+ 101us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:07.519195 (+ 209us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:07.519382 (+ 187us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:07.519438 (+ 56us) write_op.cc:312] APPLY: finished
1211 08:58:07.519632 (+ 194us) log.cc:844] Serialized 23 byte log entry
1211 08:58:07.519784 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.519862 (+ 78us) write_op.cc:454] Released schema lock
1211 08:58:07.519912 (+ 50us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":29788,"reported_tablets":30,"spinlock_wait_cycles":761394176,"child_traces":[["op",{"apply.queue_time_us":172,"apply.run_cpu_time_us":1398,"apply.run_wall_time_us":1397,"num_ops":1,"prepare.queue_time_us":31879,"prepare.run_cpu_time_us":2581,"prepare.run_wall_time_us":4036,"raft.queue_time_us":84,"raft.run_cpu_time_us":740,"raft.run_wall_time_us":740,"replication_time_us":9644}]]}
W20251211 08:58:07.739743 14007 rpcz_store.cc:269] 1211 08:58:04.860455 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:04.860563 (+ 108us) service_pool.cc:225] Handling call
1211 08:58:07.394118 (+2533555us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.394140 (+ 22us) write_op.cc:454] Released schema lock
1211 08:58:07.724518 (+330378us) spinlock_profiling.cc:243] Waited 32.2 ms on lock 0x7b4c00002680. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20991d060 00007fc20992deb2 00007fc20992e412 00007fc2098f2e7b 00007fc20992057c 00007fc209920a3e 00007fc20b286c1b 00007fc20d9d3fc1 00007fc20d97c140 00007fc20d98221b 00007fc20da85d0e
1211 08:58:07.739156 (+ 14638us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:06.428226 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:06.428701 (+ 475us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:06.428728 (+ 27us) write_op.cc:435] Acquired schema lock
1211 08:58:06.428745 (+ 17us) tablet.cc:660] Decoding operations
1211 08:58:06.429089 (+ 344us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:06.429143 (+ 54us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:06.429173 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:06.429447 (+ 274us) tablet.cc:699] Row locks acquired
1211 08:58:06.429463 (+ 16us) write_op.cc:260] PREPARE: finished
1211 08:58:06.429581 (+ 118us) write_op.cc:270] Start()
1211 08:58:06.429672 (+ 91us) write_op.cc:276] Timestamp: P: 1765443486429657 usec, L: 0
1211 08:58:06.429690 (+ 18us) op_driver.cc:348] REPLICATION: starting
1211 08:58:06.430226 (+ 536us) log.cc:844] Serialized 907 byte log entry
1211 08:58:06.731545 (+301319us) op_driver.cc:464] REPLICATION: finished
1211 08:58:06.831782 (+100237us) write_op.cc:301] APPLY: starting
1211 08:58:06.833024 (+ 1242us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:06.833158 (+ 134us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:06.833248 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:06.833520 (+ 272us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:06.833651 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1211 08:58:06.833695 (+ 44us) write_op.cc:312] APPLY: finished
1211 08:58:06.833939 (+ 244us) log.cc:844] Serialized 29 byte log entry
1211 08:58:06.834074 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:06.834136 (+ 62us) write_op.cc:454] Released schema lock
1211 08:58:06.834164 (+ 28us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:06.870802 (+ 36638us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:06.870827 (+ 25us) write_op.cc:454] Released schema lock
Related trace 'op':
1211 08:58:06.842983 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:06.843422 (+ 439us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:06.843452 (+ 30us) write_op.cc:435] Acquired schema lock
1211 08:58:06.843463 (+ 11us) tablet.cc:660] Decoding operations
1211 08:58:06.843725 (+ 262us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:06.843766 (+ 41us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:06.843790 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:06.843939 (+ 149us) tablet.cc:699] Row locks acquired
1211 08:58:06.843951 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:58:06.844061 (+ 110us) write_op.cc:270] Start()
1211 08:58:06.844145 (+ 84us) write_op.cc:276] Timestamp: P: 1765443486844088 usec, L: 0
1211 08:58:06.844162 (+ 17us) op_driver.cc:348] REPLICATION: starting
1211 08:58:06.844684 (+ 522us) log.cc:844] Serialized 356 byte log entry
1211 08:58:06.878436 (+ 33752us) op_driver.cc:464] REPLICATION: finished
1211 08:58:06.885617 (+ 7181us) write_op.cc:301] APPLY: starting
1211 08:58:06.885680 (+ 63us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:06.885763 (+ 83us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:06.885845 (+ 82us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:06.885987 (+ 142us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:06.886103 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:06.886147 (+ 44us) write_op.cc:312] APPLY: finished
1211 08:58:06.886312 (+ 165us) log.cc:844] Serialized 23 byte log entry
1211 08:58:06.886473 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:06.886532 (+ 59us) write_op.cc:454] Released schema lock
1211 08:58:06.886559 (+ 27us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:06.915702 (+ 29143us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:06.915738 (+ 36us) write_op.cc:454] Released schema lock
Related trace 'op':
1211 08:58:07.380070 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:07.380526 (+ 456us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:07.380557 (+ 31us) write_op.cc:435] Acquired schema lock
1211 08:58:07.380575 (+ 18us) tablet.cc:660] Decoding operations
1211 08:58:07.380842 (+ 267us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:07.380881 (+ 39us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:07.380902 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:07.381060 (+ 158us) tablet.cc:699] Row locks acquired
1211 08:58:07.381070 (+ 10us) write_op.cc:260] PREPARE: finished
1211 08:58:07.381172 (+ 102us) write_op.cc:270] Start()
1211 08:58:07.381244 (+ 72us) write_op.cc:276] Timestamp: P: 1765443487381224 usec, L: 0
1211 08:58:07.381256 (+ 12us) op_driver.cc:348] REPLICATION: starting
1211 08:58:07.381704 (+ 448us) log.cc:844] Serialized 356 byte log entry
1211 08:58:07.388813 (+ 7109us) op_driver.cc:464] REPLICATION: finished
1211 08:58:07.389241 (+ 428us) write_op.cc:301] APPLY: starting
1211 08:58:07.389288 (+ 47us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:07.389367 (+ 79us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:07.389445 (+ 78us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:07.389591 (+ 146us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:07.389694 (+ 103us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:07.389729 (+ 35us) write_op.cc:312] APPLY: finished
1211 08:58:07.389869 (+ 140us) log.cc:844] Serialized 23 byte log entry
1211 08:58:07.390007 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.390061 (+ 54us) write_op.cc:454] Released schema lock
1211 08:58:07.390089 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":62514,"reported_tablets":32,"spinlock_wait_cycles":74005376,"child_traces":[["op",{"apply.queue_time_us":100058,"apply.run_cpu_time_us":1363,"apply.run_wall_time_us":38969,"num_ops":2,"prepare.queue_time_us":8258,"prepare.run_cpu_time_us":2376,"prepare.run_wall_time_us":48658,"raft.queue_time_us":44466,"raft.run_cpu_time_us":667,"raft.run_wall_time_us":667,"replication_time_us":301748,"thread_start_us":70077,"threads_started":2}],["op",{"apply.queue_time_us":7026,"apply.run_cpu_time_us":1182,"apply.run_wall_time_us":30028,"num_ops":1,"prepare.queue_time_us":5522,"prepare.run_cpu_time_us":2007,"prepare.run_wall_time_us":2005,"raft.queue_time_us":9363,"raft.run_cpu_time_us":653,"raft.run_wall_time_us":1338,"replication_time_us":34181}],["op",{"apply.queue_time_us":302,"apply.run_cpu_time_us":1004,"apply.run_wall_time_us":1002,"num_ops":1,"prepare.queue_time_us":99,"prepare.run_cpu_time_us":1893,"prepare.run_wall_time_us":1892,"raft.queue_time_us":189,"raft.run_cpu_time_us":574,"raft.run_wall_time_us":574,"replication_time_us":7460}]]}
W20251211 08:58:07.787710 14000 connection.cc:537] server connection from 127.13.146.1:34545 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:07.742046 14012 rpcz_store.cc:269] 1211 08:58:05.955042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:05.955169 (+ 127us) service_pool.cc:225] Handling call
1211 08:58:07.739220 (+1784051us) spinlock_profiling.cc:243] Waited 49.7 ms on lock 0x7b4c00002680. stack: 00007fc204ba8083 00007fc204ba7f80 00007fc20471d574 0000000000508744 00000000005086d9 00007fc20991d060 00007fc20992deb2 00007fc20992e412 00007fc2098f2e7b 00007fc20992057c 00007fc209920a3e 00007fc20b286c1b 00007fc20d9d3fc1 00007fc20d97c140 00007fc20d98221b 00007fc20da85d0e
1211 08:58:07.741674 (+ 2454us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:07.321460 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:07.321922 (+ 462us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:07.321950 (+ 28us) write_op.cc:435] Acquired schema lock
1211 08:58:07.321986 (+ 36us) tablet.cc:660] Decoding operations
1211 08:58:07.322250 (+ 264us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:07.322291 (+ 41us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:07.322314 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
1211 08:58:07.322459 (+ 145us) tablet.cc:699] Row locks acquired
1211 08:58:07.322471 (+ 12us) write_op.cc:260] PREPARE: finished
1211 08:58:07.322583 (+ 112us) write_op.cc:270] Start()
1211 08:58:07.322665 (+ 82us) write_op.cc:276] Timestamp: P: 1765443487322605 usec, L: 0
1211 08:58:07.322679 (+ 14us) op_driver.cc:348] REPLICATION: starting
1211 08:58:07.323275 (+ 596us) log.cc:844] Serialized 356 byte log entry
1211 08:58:07.331138 (+ 7863us) op_driver.cc:464] REPLICATION: finished
1211 08:58:07.331436 (+ 298us) write_op.cc:301] APPLY: starting
1211 08:58:07.331495 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:07.331585 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:07.331689 (+ 104us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:07.331870 (+ 181us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:07.332007 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1211 08:58:07.332052 (+ 45us) write_op.cc:312] APPLY: finished
1211 08:58:07.332215 (+ 163us) log.cc:844] Serialized 23 byte log entry
1211 08:58:07.332352 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.332409 (+ 57us) write_op.cc:454] Released schema lock
1211 08:58:07.332440 (+ 31us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:07.332652 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:07.332673 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":60834,"reported_tablets":32,"spinlock_wait_cycles":114377088,"child_traces":[["op",{"apply.queue_time_us":164,"apply.run_cpu_time_us":1185,"apply.run_wall_time_us":1183,"num_ops":1,"prepare.queue_time_us":35286,"prepare.run_cpu_time_us":2138,"prepare.run_wall_time_us":2137,"raft.queue_time_us":187,"raft.run_cpu_time_us":640,"raft.run_wall_time_us":639,"replication_time_us":8367}]]}
W20251211 08:58:07.795807 14544 consensus_peers.cc:597] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:07.796692 14544 consensus_peers.cc:597] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:07.797112 14544 leader_election.cc:336] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111)
W20251211 08:58:07.797875 14544 leader_election.cc:336] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111)
W20251211 08:58:07.800227 14000 catalog_manager.cc:4712] TS 31404d5f58e548519239569b150d975b (127.13.146.1:36181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c02cb6ec01041dba118683a57aecde2: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111)
W20251211 08:58:07.798523 14544 consensus_peers.cc:597] T d4f826d079774152a3b66bfb81f990a8 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:07.800948 14544 consensus_peers.cc:597] T f5c5b1aebde94dea81cb37a6a6998070 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:07.806994 14544 consensus_peers.cc:597] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:07.807533 14544 consensus_peers.cc:597] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:07.807909 14544 consensus_peers.cc:597] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:07.819123 14544 consensus_peers.cc:597] T 8c39cc816f4c438391bb4826ea1a508f P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:07.822398 14544 consensus_peers.cc:597] T 825d8cf806d94702bdd11e0d63ee0c59 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:07.808634 15030 tablet_bootstrap.cc:492] T 4c95565ce3954cc598d7cbd45321d7c9 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:07.856498 15030 tablet_bootstrap.cc:654] T 4c95565ce3954cc598d7cbd45321d7c9 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:07.860850 16491 ts_tablet_manager.cc:1929] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:07.861347 16491 log.cc:1199] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/07f983604bfa474f9c9ce0c982d738fc
I20251211 08:58:07.862218 16491 ts_tablet_manager.cc:1950] T 07f983604bfa474f9c9ce0c982d738fc P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:07.864415 16301 raft_consensus.cc:493] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:07.864825 16491 tablet_replica.cc:333] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:07.864863 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 07f983604bfa474f9c9ce0c982d738fc (table default.Table_80758110ce794260af2a551849bf4896 [id=277e1a7e34f54c16b29a738c17f75da5]) successfully deleted
I20251211 08:58:07.865646 16491 raft_consensus.cc:2243] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:07.866076 16491 raft_consensus.cc:2272] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:07.864826 16301 raft_consensus.cc:515] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.876704 16301 leader_election.cc:290] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31404d5f58e548519239569b150d975b (127.13.146.1:36181), b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153)
W20251211 08:58:07.888446 14544 leader_election.cc:336] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111)
I20251211 08:58:07.888676 14573 tablet_service.cc:1505] Processing CreateTablet for tablet 36a3f351f4f049e8a122c76d9b1f33ff (DEFAULT_TABLE table=default.Table_7ff5b865284f49c6826753de45156314 [id=c589ea1ae2d74f839cdabccf5a953a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:07.891420 14586 tablet_service.cc:1505] Processing CreateTablet for tablet 7225ab3706c6482ab93f8d16fd208c75 (DEFAULT_TABLE table=default.Table_835755f0e12346f38507b3c844df43da [id=48ca7d885ff745a5acfdf1d259736bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:07.893095 14576 tablet_service.cc:1505] Processing CreateTablet for tablet 8c6396097c934310b7c44f73517aad7c (DEFAULT_TABLE table=default.Table_07f6651621854d2f848120541ef54d4c [id=8ce415ba0c814346ab9e7c555348d50d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:07.894737 14580 tablet_service.cc:1505] Processing CreateTablet for tablet 55b061c23b374318ac72c6c6d6b86acc (DEFAULT_TABLE table=default.Table_52f6bbd3c66a47e7949baa7892ea2e67 [id=814453ce1f224591b0226e2e37965987]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:07.897434 14574 tablet_service.cc:1505] Processing CreateTablet for tablet 5f9e1897ff5445a9b9b513d95273f64a (DEFAULT_TABLE table=default.Table_210d92e56fb84636a1756a4f3c356e21 [id=c11fe30876af49e5a4818edefd437c6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:07.898384 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 4cbd18e927c843e7b0aee7e0ccb1b06d (table default.Table_d458d069d7d54a4f818a89e76f6d658b [id=a9e8baf63d354fdaab9fb2a0381ffbcd]). Expected version 1 got 0
I20251211 08:58:07.901162 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet d7a8acb51a2948f4a018787bfaecb694 (table default.Table_9e007a9469d54d99b9471cb8e653b620 [id=de3a242b7e6d480ebf509e458ec002c7]). Expected version 1 got 0
I20251211 08:58:07.901212 14587 tablet_service.cc:1505] Processing CreateTablet for tablet e01d89ad1618494fae618112f430a629 (DEFAULT_TABLE table=default.Table_7ca1c0fc20ca4dd7a11793c0ac968666 [id=67ab831326db412fa228aac2683950ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:07.902381 14005 catalog_manager.cc:5654] T 7264e13dd44b499683e32b3fc62ed8ed P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20251211 08:58:07.915254 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 552de4c6d030418181f0400714d9ec31 (table default.Table_3e1d63cec2a5456788ac674e54f4669f [id=15e81fa14b7147ff8afe4446db15c00e]). Expected version 1 got 0
I20251211 08:58:07.916558 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
I20251211 08:58:07.917544 14005 catalog_manager.cc:5654] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } }
I20251211 08:58:07.920329 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 4dcd09f7c6364d74bb33503b5bbfea96 (table default.Table_78e567c0deac4f8d99c850bc335bbaf5 [id=482af2b30df84c8891e037e1aea436af]). Expected version 1 got 0
I20251211 08:58:07.924208 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:07.924981 14005 catalog_manager.cc:5654] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: HEALTHY } } }
I20251211 08:58:07.928328 14573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36a3f351f4f049e8a122c76d9b1f33ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251211 08:58:07.929571 14544 consensus_peers.cc:597] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:07.932088 15030 tablet_bootstrap.cc:492] T 4c95565ce3954cc598d7cbd45321d7c9 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:07.932471 15030 ts_tablet_manager.cc:1403] T 4c95565ce3954cc598d7cbd45321d7c9 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.124s user 0.001s sys 0.007s
W20251211 08:58:07.940099 14544 consensus_peers.cc:597] T 111adfd1ac1a42bdafa9416594e7e932 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:07.940021 15030 raft_consensus.cc:359] T 4c95565ce3954cc598d7cbd45321d7c9 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.940552 15030 raft_consensus.cc:385] T 4c95565ce3954cc598d7cbd45321d7c9 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:07.940769 15030 raft_consensus.cc:740] T 4c95565ce3954cc598d7cbd45321d7c9 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:07.941247 15030 consensus_queue.cc:260] T 4c95565ce3954cc598d7cbd45321d7c9 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:07.944147 14586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7225ab3706c6482ab93f8d16fd208c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:07.947209 14576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c6396097c934310b7c44f73517aad7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:07.952234 14580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55b061c23b374318ac72c6c6d6b86acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:07.955231 16491 ts_tablet_manager.cc:1916] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:07.955273 14574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f9e1897ff5445a9b9b513d95273f64a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:07.963039 14587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01d89ad1618494fae618112f430a629. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:07.963078 15030 ts_tablet_manager.cc:1434] T 4c95565ce3954cc598d7cbd45321d7c9 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.030s user 0.004s sys 0.002s
I20251211 08:58:07.963915 15030 tablet_bootstrap.cc:492] T 1065085c701f48c9ad7206c410ce5985 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
W20251211 08:58:07.978147 14544 consensus_peers.cc:597] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:07.990482 15030 tablet_bootstrap.cc:654] T 1065085c701f48c9ad7206c410ce5985 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:07.991775 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet e36c6e2868e143be8e7f72307202a268 (table default.Table_7cf998dd4bd84149b73b791cff8c3652 [id=8f146b956cf348b49a4d9694bfaca344]). Expected version 2 got 0
I20251211 08:58:08.004705 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 680c8f54e75a4291bb4d769d94e21552 (table default.Table_032686fd837d4292bd6c95749e67b5de [id=387069b529984870961fe2e545fbb221]). Expected version 2 got 0
I20251211 08:58:08.007041 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet d7a8acb51a2948f4a018787bfaecb694 (table default.Table_9e007a9469d54d99b9471cb8e653b620 [id=de3a242b7e6d480ebf509e458ec002c7]). Expected version 1 got 0
I20251211 08:58:08.007754 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet c0bb339322504704ae57bba2f4c3a9bb (table default.Table_0e736d3e3e5f40dbbf25a5e615e5b842 [id=f48fec067bf941a4a62265c36fe526da]). Expected version 1 got 0
I20251211 08:58:08.009097 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 9cc3373962574253868ababbbe78e57d (table default.Table_2ff48e3e6b214a0b8711f3bc10ed9319 [id=2c2c410df07d4e4cbc60e08dce22efe9]). Expected version 2 got 0
I20251211 08:58:08.009701 14010 catalog_manager.cc:5746] TS b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153) does not have the latest schema for tablet 4dcd09f7c6364d74bb33503b5bbfea96 (table default.Table_78e567c0deac4f8d99c850bc335bbaf5 [id=482af2b30df84c8891e037e1aea436af]). Expected version 1 got 0
I20251211 08:58:08.016358 16491 ts_tablet_manager.cc:1929] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:08.016707 16491 log.cc:1199] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/5d8d84e594ea47f3bc9127cc587e8aa7
I20251211 08:58:08.017306 16491 ts_tablet_manager.cc:1950] T 5d8d84e594ea47f3bc9127cc587e8aa7 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
I20251211 08:58:08.020200 15030 tablet_bootstrap.cc:492] T 1065085c701f48c9ad7206c410ce5985 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:08.020551 15030 ts_tablet_manager.cc:1403] T 1065085c701f48c9ad7206c410ce5985 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.000s
I20251211 08:58:08.033816 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 07f983604bfa474f9c9ce0c982d738fc with delete_type TABLET_DATA_DELETED (replaced by tablet 960bc7a27fa04f2c9f398a67d2c93266) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:08.041895 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet 5d8d84e594ea47f3bc9127cc587e8aa7 (table default.Table_70f3ebdd01dc4e75bc85ab29528865fc [id=67d1517b43294838bda420b792edb9c5]) successfully deleted
I20251211 08:58:08.038982 15030 raft_consensus.cc:359] T 1065085c701f48c9ad7206c410ce5985 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:08.043371 15030 raft_consensus.cc:385] T 1065085c701f48c9ad7206c410ce5985 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:08.043680 15030 raft_consensus.cc:740] T 1065085c701f48c9ad7206c410ce5985 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:08.044379 15030 consensus_queue.cc:260] T 1065085c701f48c9ad7206c410ce5985 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } }
I20251211 08:58:08.046991 15030 ts_tablet_manager.cc:1434] T 1065085c701f48c9ad7206c410ce5985 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20251211 08:58:08.047894 15030 tablet_bootstrap.cc:492] T 061e2b441d9c4b03b12d8ed31dce9595 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:08.042158 16491 tablet_replica.cc:333] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
I20251211 08:58:08.078166 14574 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d8d84e594ea47f3bc9127cc587e8aa7 with delete_type TABLET_DATA_DELETED (replaced by tablet e62ae5b540aa4680b4d18fee29c39e64) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:08.078837 14580 tablet_service.cc:1552] Processing DeleteTablet for tablet 7af7c9ab27a64a1681f9874a922e986f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-11 08:57:42 UTC) from {username='slave'} at 127.0.0.1:42846
I20251211 08:58:08.079015 16491 raft_consensus.cc:2243] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:08.079982 16491 raft_consensus.cc:2272] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:08.082001 16491 ts_tablet_manager.cc:1916] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:08.089488 15030 tablet_bootstrap.cc:654] T 061e2b441d9c4b03b12d8ed31dce9595 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:08.108445 15030 tablet_bootstrap.cc:492] T 061e2b441d9c4b03b12d8ed31dce9595 P c5e624bcfd154cd586ef1dedb9940063: No bootstrap required, opened a new log
I20251211 08:58:08.108877 15030 ts_tablet_manager.cc:1403] T 061e2b441d9c4b03b12d8ed31dce9595 P c5e624bcfd154cd586ef1dedb9940063: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.000s
I20251211 08:58:08.111272 15030 raft_consensus.cc:359] T 061e2b441d9c4b03b12d8ed31dce9595 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:08.111909 15030 raft_consensus.cc:385] T 061e2b441d9c4b03b12d8ed31dce9595 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:08.112154 15030 raft_consensus.cc:740] T 061e2b441d9c4b03b12d8ed31dce9595 P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e624bcfd154cd586ef1dedb9940063, State: Initialized, Role: FOLLOWER
I20251211 08:58:08.112818 15030 consensus_queue.cc:260] T 061e2b441d9c4b03b12d8ed31dce9595 P c5e624bcfd154cd586ef1dedb9940063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } }
I20251211 08:58:08.114842 15030 ts_tablet_manager.cc:1434] T 061e2b441d9c4b03b12d8ed31dce9595 P c5e624bcfd154cd586ef1dedb9940063: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251211 08:58:08.115590 15030 tablet_bootstrap.cc:492] T f56fd93aeac6435ea84388ce05580113 P c5e624bcfd154cd586ef1dedb9940063: Bootstrap starting.
I20251211 08:58:08.125021 16491 ts_tablet_manager.cc:1929] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:08.125342 16491 log.cc:1199] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/ac6be204692a44fab2db551e79237299
I20251211 08:58:08.125969 16491 ts_tablet_manager.cc:1950] T ac6be204692a44fab2db551e79237299 P c5e624bcfd154cd586ef1dedb9940063: Deleting consensus metadata
W20251211 08:58:08.132076 14010 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.13.146.2:44835 (request call id 69) took 1060 ms (client timeout 1000 ms). Trace:
W20251211 08:58:08.132377 14010 rpcz_store.cc:269] 1211 08:58:07.071760 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:07.420220 (+348460us) service_pool.cc:225] Handling call
1211 08:58:08.132010 (+711790us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22866,"reported_tablets":35}
I20251211 08:58:08.133072 15030 tablet_bootstrap.cc:654] T f56fd93aeac6435ea84388ce05580113 P c5e624bcfd154cd586ef1dedb9940063: Neither blocks nor log segments found. Creating new log.
W20251211 08:58:08.166683 14542 connection.cc:537] client connection to 127.13.146.2:36153 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:08.177484 13998 connection.cc:537] client connection to 127.13.146.2:36153 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251211 08:58:08.178819 13997 catalog_manager.cc:4985] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291): tablet ac6be204692a44fab2db551e79237299 (table default.Table_fe31f77461b643beb43b390d2e857446 [id=493f7d8563d940a38d7a549cbce5e93e]) successfully deleted
W20251211 08:58:08.179214 16497 negotiation.cc:337] Failed RPC negotiation. Trace:
1211 08:58:08.170539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.146.2:36153 (local address 127.13.146.3:46619)
1211 08:58:08.171940 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
1211 08:58:08.172007 (+ 67us) client_negotiation.cc:174] Beginning negotiation
1211 08:58:08.172444 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1211 08:58:08.178342 (+ 5898us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.146.2:36153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":829}
I20251211 08:58:08.178599 16491 tablet_replica.cc:333] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063: stopping tablet replica
W20251211 08:58:08.179648 14542 consensus_peers.cc:597] T 41d6ef975a47489385791065bb8f9f46 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.2:36153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:08.180078 14542 leader_election.cc:336] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Network error: Client connection negotiation failed: client connection to 127.13.146.2:36153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251211 08:58:08.180418 14542 leader_election.cc:304] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e624bcfd154cd586ef1dedb9940063; no voters: 31404d5f58e548519239569b150d975b, b276f77f87254a5d909c5ab940a39256
I20251211 08:58:08.180578 16491 raft_consensus.cc:2243] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:08.181010 16491 raft_consensus.cc:2272] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:08.181020 16412 raft_consensus.cc:2749] T a74d250ee85a4c768c76a8400003ccdb P c5e624bcfd154cd586ef1dedb9940063 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 08:58:08.181792 14542 consensus_peers.cc:597] T b5884dc6ef814552893cdf9dccfcce19 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.2:36153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:08.182106 14542 consensus_peers.cc:597] T 4dcd09f7c6364d74bb33503b5bbfea96 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.2:36153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:08.182426 14542 consensus_peers.cc:597] T 9cc3373962574253868ababbbe78e57d P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.2:36153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:08.182756 14542 consensus_peers.cc:597] T 559f9d428b5841708999725b5b859660 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.2:36153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:08.182832 16491 ts_tablet_manager.cc:1916] T 476eee1606e74832abf928dbaafc844d P c5e624bcfd154cd586ef1dedb9940063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:58:08.200575 14544 consensus_peers.cc:597] T 363fe7a2e1ac4fb4a1ec97dc2354d71a P c5e624bcfd154cd586ef1dedb9940063 -> Peer 31404d5f58e548519239569b150d975b (127.13.146.1:36181): Couldn't send request to peer 31404d5f58e548519239569b150d975b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.1:36181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:08.207118 14542 consensus_peers.cc:597] T bfb3a2b8f8f64c12ab590ffc0be7ace3 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.2:36153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:08.207612 14542 consensus_peers.cc:597] T 88860fa9d7e6414185f0a1cbc78b2571 P c5e624bcfd154cd586ef1dedb9940063 -> Peer b276f77f87254a5d909c5ab940a39256 (127.13.146.2:36153): Couldn't send request to peer b276f77f87254a5d909c5ab940a39256. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.2:36153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251211 08:58:08.208868 14000 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 48575fa3f45349b9887bfc1cdfebc958 (table default.Table_a7ed3d0a6ce0464893bc9f5bb3bf6851 [id=1b3efd8519a04b13af86aa631f596f33], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 48575fa3f45349b9887bfc1cdfebc958 consensus state has no leader
I20251211 08:58:08.209697 13896 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu with pid 14505
W20251211 08:58:08.213821 14000 connection.cc:537] server connection from 127.13.146.2:44835 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251211 08:58:08.277649 14005 catalog_manager.cc:5746] TS c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3:37291) does not have the latest schema for tablet 680c8f54e75a4291bb4d769d94e21552 (table default.Table_032686fd837d4292bd6c95749e67b5de [id=387069b529984870961fe2e545fbb221]). Expected version 2 got 0
I20251211 08:58:08.278463 14005 catalog_manager.cc:5654] T 947e6203f910442ca01e2c21587cfe90 P c5e624bcfd154cd586ef1dedb9940063 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e624bcfd154cd586ef1dedb9940063 (127.13.146.3). New cstate: current_term: 1 leader_uuid: "c5e624bcfd154cd586ef1dedb9940063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31404d5f58e548519239569b150d975b" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e624bcfd154cd586ef1dedb9940063" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b276f77f87254a5d909c5ab940a39256" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20251211 08:58:08.393455 16505 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4c9cd45be3947d09423ed4508ca2e91)
I20251211 08:58:08.393887 16505 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73702d54989f4959acfc4a6fdcede6ca" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 39015 } } peers { permanent_uuid: "a4c9cd45be3947d09423ed4508ca2e91" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 34977 } } peers { permanent_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 38671 } }
I20251211 08:58:08.396987 14023 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 596 } } ignore_live_leader: false dest_uuid: "a4c9cd45be3947d09423ed4508ca2e91" is_pre_election: true
I20251211 08:58:08.403098 16505 leader_election.cc:290] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4c9cd45be3947d09423ed4508ca2e91 (127.13.146.61:34977), 96e37aef4ef94e33a8a6807c7a9c404f (127.13.146.60:38671)
I20251211 08:58:08.403239 14104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73702d54989f4959acfc4a6fdcede6ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 596 } } ignore_live_leader: false dest_uuid: "96e37aef4ef94e33a8a6807c7a9c404f" is_pre_election: true
I20251211 08:58:08.404690 15726 leader_election.cc:304] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73702d54989f4959acfc4a6fdcede6ca; no voters: 96e37aef4ef94e33a8a6807c7a9c404f, a4c9cd45be3947d09423ed4508ca2e91
I20251211 08:58:08.412346 16505 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 73702d54989f4959acfc4a6fdcede6ca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251211 08:58:08.590423 14000 connection.cc:537] server connection from 127.13.146.3:43327 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:08.590900 13997 connection.cc:537] client connection to 127.13.146.3:37291 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251211 08:58:08.624002 13896 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu with pid 14931
I20251211 08:58:08.703979 13896 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu with pid 13975
I20251211 08:58:08.768429 13896 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu with pid 14050
2025-12-11T08:58:08Z chronyd exiting
I20251211 08:58:08.821545 13896 master-stress-test.cc:585] Tables created: 330
I20251211 08:58:08.821722 13896 master-stress-test.cc:586] Tables altered: 149
I20251211 08:58:08.821846 13896 master-stress-test.cc:587] Tables deleted: 122
I20251211 08:58:08.822001 13896 master-stress-test.cc:588] Tablets replaced: 54
I20251211 08:58:08.822142 13896 master-stress-test.cc:589] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (62210 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
I20251211 08:58:08.937914 13896 test_util.cc:276] Using random seed: 1476884386
2025-12-11T08:58:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-11T08:58:08Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskt39ak2/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskt39ak2/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/security.properties
2025-12-11 08:58:14: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskt39ak2/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskt39ak2/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20251211 08:58:22.330379 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.62:43161
--webserver_interface=127.13.146.62
--webserver_port=0
--builtin_ntp_servers=127.13.146.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--hive_metastore_uris=thrift://127.0.0.1:46123
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251211 08:58:22.620478 16744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:58:22.621073 16744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:58:22.621516 16744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:58:22.649426 16744 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20251211 08:58:22.649619 16744 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20251211 08:58:22.649780 16744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251211 08:58:22.649947 16744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:58:22.650111 16744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251211 08:58:22.650272 16744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251211 08:58:22.681917 16744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:46123
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.62:43161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.146.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:58:22.683079 16744 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:58:22.684533 16744 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:58:22.693778 16750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:24.098376 16749 debug-util.cc:398] Leaking SignalData structure 0x7b0800028120 after lost signal to thread 16744
W20251211 08:58:24.412050 16749 kernel_stack_watchdog.cc:198] Thread 16744 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20251211 08:58:22.695379 16751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:24.412734 16744 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.718s user 0.002s sys 0.000s
W20251211 08:58:24.413130 16744 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.718s user 0.002s sys 0.000s
W20251211 08:58:24.413352 16752 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1718 milliseconds
W20251211 08:58:24.415023 16753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:24.414976 16744 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251211 08:58:24.417881 16744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:58:24.420872 16744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:58:24.422411 16744 hybrid_clock.cc:648] HybridClock initialized: now 1765443504422356 us; error 62 us; skew 500 ppm
I20251211 08:58:24.423262 16744 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:58:24.429350 16744 webserver.cc:492] Webserver started at http://127.13.146.62:33501/ using document root <none> and password file <none>
I20251211 08:58:24.430244 16744 fs_manager.cc:362] Metadata directory not provided
I20251211 08:58:24.430436 16744 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:58:24.430927 16744 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:58:24.435083 16744 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/data/instance:
uuid: "133269af8ce44b579ebbb7805bc58800"
format_stamp: "Formatted at 2025-12-11 08:58:24 on dist-test-slave-0dfs"
I20251211 08:58:24.436080 16744 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/wal/instance:
uuid: "133269af8ce44b579ebbb7805bc58800"
format_stamp: "Formatted at 2025-12-11 08:58:24 on dist-test-slave-0dfs"
I20251211 08:58:24.442589 16744 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20251211 08:58:24.447938 16760 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:24.448889 16744 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20251211 08:58:24.449189 16744 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/wal
uuid: "133269af8ce44b579ebbb7805bc58800"
format_stamp: "Formatted at 2025-12-11 08:58:24 on dist-test-slave-0dfs"
I20251211 08:58:24.449487 16744 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:58:24.511278 16744 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:58:24.513129 16744 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:58:24.513584 16744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:58:24.580819 16744 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.62:43161
I20251211 08:58:24.580884 16811 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.62:43161 every 8 connection(s)
I20251211 08:58:24.583470 16744 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/data/info.pb
I20251211 08:58:24.584337 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 16744
I20251211 08:58:24.584715 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-0/wal/instance
I20251211 08:58:24.587101 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.61:36035
--webserver_interface=127.13.146.61
--webserver_port=0
--builtin_ntp_servers=127.13.146.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--hive_metastore_uris=thrift://127.0.0.1:46123
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251211 08:58:24.589314 16812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:24.599033 16812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:58:24.627339 16812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } has no permanent_uuid. Determining permanent_uuid...
W20251211 08:58:24.629266 16761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
W20251211 08:58:24.633111 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:36035: Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:24.682634 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } attempt: 1
W20251211 08:58:24.686283 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:36035: Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:24.754797 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } attempt: 2
W20251211 08:58:24.758445 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:36035: Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:24.849897 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } attempt: 3
W20251211 08:58:24.853578 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:36035: Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
W20251211 08:58:24.879230 16813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:58:24.879758 16813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:58:24.880143 16813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:58:24.909849 16813 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20251211 08:58:24.910069 16813 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20251211 08:58:24.910245 16813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251211 08:58:24.910416 16813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:58:24.910596 16813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251211 08:58:24.910785 16813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251211 08:58:24.944854 16813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:46123
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.61:36035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.146.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:58:24.946038 16813 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:58:24.947683 16813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:58:24.956947 16822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:24.997090 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } attempt: 4
W20251211 08:58:25.004133 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:36035: Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:25.303632 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } attempt: 5
W20251211 08:58:25.310254 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:36035: Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:25.857815 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } attempt: 6
W20251211 08:58:25.888139 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.61:36035: Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
W20251211 08:58:24.958593 16823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:26.336405 16825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:26.338613 16824 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1375 milliseconds
W20251211 08:58:26.338869 16813 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.381s user 0.429s sys 0.929s
W20251211 08:58:26.339221 16813 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.381s user 0.433s sys 0.929s
I20251211 08:58:26.339435 16813 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251211 08:58:26.340503 16813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:58:26.343025 16813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:58:26.344380 16813 hybrid_clock.cc:648] HybridClock initialized: now 1765443506344351 us; error 38 us; skew 500 ppm
I20251211 08:58:26.345098 16813 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:58:26.352572 16813 webserver.cc:492] Webserver started at http://127.13.146.61:41841/ using document root <none> and password file <none>
I20251211 08:58:26.353525 16813 fs_manager.cc:362] Metadata directory not provided
I20251211 08:58:26.353735 16813 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:58:26.354139 16813 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:58:26.358207 16813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data/instance:
uuid: "c3112fc6f4ae4541a2af2405a97c707b"
format_stamp: "Formatted at 2025-12-11 08:58:26 on dist-test-slave-0dfs"
I20251211 08:58:26.359270 16813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal/instance:
uuid: "c3112fc6f4ae4541a2af2405a97c707b"
format_stamp: "Formatted at 2025-12-11 08:58:26 on dist-test-slave-0dfs"
I20251211 08:58:26.366624 16813 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20251211 08:58:26.372545 16834 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:26.373770 16813 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20251211 08:58:26.374083 16813 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal
uuid: "c3112fc6f4ae4541a2af2405a97c707b"
format_stamp: "Formatted at 2025-12-11 08:58:26 on dist-test-slave-0dfs"
I20251211 08:58:26.374410 16813 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:58:26.443308 16813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:58:26.444613 16813 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:58:26.445022 16813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:58:26.510932 16813 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.61:36035
I20251211 08:58:26.510974 16885 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.61:36035 every 8 connection(s)
I20251211 08:58:26.513496 16813 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data/info.pb
I20251211 08:58:26.515699 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 16813
I20251211 08:58:26.516212 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal/instance
I20251211 08:58:26.519694 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.60:34701
--webserver_interface=127.13.146.60
--webserver_port=0
--builtin_ntp_servers=127.13.146.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--hive_metastore_uris=thrift://127.0.0.1:46123
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251211 08:58:26.520134 16886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:26.533226 16886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:58:26.552969 16886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:58:26.565060 16886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } has no permanent_uuid. Determining permanent_uuid...
W20251211 08:58:26.567274 16837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 08:58:26.571027 16886 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:26.620560 16886 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 1
W20251211 08:58:26.624825 16886 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:26.693194 16886 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 2
W20251211 08:58:26.697144 16886 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:26.788513 16886 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 3
W20251211 08:58:26.792680 16886 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 08:58:26.825745 16887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:58:26.826438 16887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:58:26.826957 16887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:58:26.857378 16887 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20251211 08:58:26.857613 16887 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20251211 08:58:26.857849 16887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251211 08:58:26.858125 16887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:58:26.858440 16887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251211 08:58:26.858708 16887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251211 08:58:26.892702 16887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:46123
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.60:34701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.146.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:58:26.894039 16887 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:58:26.895769 16887 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:58:26.905454 16897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:26.936208 16886 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 4
W20251211 08:58:26.942379 16886 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:26.948712 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } attempt: 7
I20251211 08:58:26.978401 16812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } has no permanent_uuid. Determining permanent_uuid...
W20251211 08:58:27.001842 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:27.060380 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 1
W20251211 08:58:27.066994 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:27.148530 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 2
W20251211 08:58:27.155230 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:27.240728 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 3
I20251211 08:58:27.241886 16886 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 5
W20251211 08:58:27.246135 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 08:58:27.246824 16886 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:27.386576 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 4
W20251211 08:58:27.393435 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:27.676946 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 5
W20251211 08:58:27.685622 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:27.794322 16886 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 6
W20251211 08:58:27.830475 16886 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:28.238170 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 6
W20251211 08:58:28.267309 16812 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 08:58:28.309774 16896 debug-util.cc:398] Leaking SignalData structure 0x7b0800028120 after lost signal to thread 16887
W20251211 08:58:28.360522 16896 kernel_stack_watchdog.cc:198] Thread 16887 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20251211 08:58:26.907178 16898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:28.361454 16887 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.455s user 0.000s sys 0.002s
W20251211 08:58:28.361778 16887 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.455s user 0.000s sys 0.003s
W20251211 08:58:28.361900 16899 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1454 milliseconds
I20251211 08:58:28.363610 16887 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20251211 08:58:28.363653 16900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:28.366058 16887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:58:28.368839 16887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:58:28.370245 16887 hybrid_clock.cc:648] HybridClock initialized: now 1765443508370206 us; error 50 us; skew 500 ppm
I20251211 08:58:28.371001 16887 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:58:28.376968 16887 webserver.cc:492] Webserver started at http://127.13.146.60:41887/ using document root <none> and password file <none>
I20251211 08:58:28.377831 16887 fs_manager.cc:362] Metadata directory not provided
I20251211 08:58:28.378013 16887 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:58:28.378436 16887 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:58:28.382519 16887 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data/instance:
uuid: "b8dd1b89d5ad4e529459606ce6f85ab8"
format_stamp: "Formatted at 2025-12-11 08:58:28 on dist-test-slave-0dfs"
I20251211 08:58:28.383553 16887 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal/instance:
uuid: "b8dd1b89d5ad4e529459606ce6f85ab8"
format_stamp: "Formatted at 2025-12-11 08:58:28 on dist-test-slave-0dfs"
I20251211 08:58:28.389992 16887 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20251211 08:58:28.395179 16915 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:28.396140 16887 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251211 08:58:28.396431 16887 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal
uuid: "b8dd1b89d5ad4e529459606ce6f85ab8"
format_stamp: "Formatted at 2025-12-11 08:58:28 on dist-test-slave-0dfs"
I20251211 08:58:28.396728 16887 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:58:28.446113 16887 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:58:28.447567 16887 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:58:28.447947 16887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:58:28.516804 16887 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.60:34701
I20251211 08:58:28.516940 16966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.60:34701 every 8 connection(s)
I20251211 08:58:28.519325 16887 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data/info.pb
I20251211 08:58:28.521652 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 16887
I20251211 08:58:28.522390 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal/instance
I20251211 08:58:28.526288 16967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:28.545198 16967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:58:28.566040 16967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:58:28.573963 16967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } has no permanent_uuid. Determining permanent_uuid...
I20251211 08:58:28.588088 16967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Bootstrap starting.
I20251211 08:58:28.594606 16967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:28.596312 16967 log.cc:826] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Log is configured to *not* fsync() on all Append() calls
I20251211 08:58:28.601482 16967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: No bootstrap required, opened a new log
I20251211 08:58:28.619343 16967 raft_consensus.cc:359] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:28.619933 16967 raft_consensus.cc:385] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:28.620190 16967 raft_consensus.cc:740] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8dd1b89d5ad4e529459606ce6f85ab8, State: Initialized, Role: FOLLOWER
I20251211 08:58:28.620854 16967 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:28.623522 16974 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:28.624112 16974 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:28.625274 16967 sys_catalog.cc:565] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: configured and running, proceeding with master startup.
W20251211 08:58:28.655498 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.655820 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:58:28.649330 16887 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: OK, instance_id { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" instance_seqno: 1765443504563048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251211 08:58:28.677659 16887 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 08:58:28.678689 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.679191 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.693449 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.693821 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.702790 16921 tablet.cc:2378] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251211 08:58:28.710484 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.710918 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.725163 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.725399 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.738063 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.738269 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.752351 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.752662 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.766733 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.767094 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.781098 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.781390 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.795559 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.795886 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.809902 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.810168 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.824432 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.824746 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.838836 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.839187 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.853255 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.853605 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.868064 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.868378 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:28.882651 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.883033 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:58:28.891006 16886 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 7
W20251211 08:58:28.897024 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.897266 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:58:28.907872 16886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Bootstrap starting.
W20251211 08:58:28.911526 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.911854 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:58:28.914131 16886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:28.915684 16886 log.cc:826] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Log is configured to *not* fsync() on all Append() calls
I20251211 08:58:28.919708 16886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: No bootstrap required, opened a new log
W20251211 08:58:28.925949 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.926242 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:58:28.939134 16974 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:28.939610 16974 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
W20251211 08:58:28.940443 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20251211 08:58:28.940136 16886 raft_consensus.cc:359] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
W20251211 08:58:28.940716 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:58:28.940904 16886 raft_consensus.cc:385] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:28.941191 16886 raft_consensus.cc:740] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3112fc6f4ae4541a2af2405a97c707b, State: Initialized, Role: FOLLOWER
I20251211 08:58:28.941869 16974 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
I20251211 08:58:28.941987 16886 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:28.944707 16993 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:28.945541 16993 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:28.946714 16886 sys_catalog.cc:565] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: configured and running, proceeding with master startup.
I20251211 08:58:28.946928 16861 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b" is_pre_election: true
I20251211 08:58:28.947660 16861 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8dd1b89d5ad4e529459606ce6f85ab8 in term 0.
I20251211 08:58:28.948846 16916 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b; no voters:
I20251211 08:58:28.949750 16974 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:28.950093 16974 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:28.950381 16974 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:58:28.952121 16840 tablet.cc:2378] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251211 08:58:28.956804 16974 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:28.960433 16974 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 1 election: Requested vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
I20251211 08:58:28.960654 16861 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b"
I20251211 08:58:28.961249 16861 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:58:28.974695 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:28.975245 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:58:28.975917 16861 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8dd1b89d5ad4e529459606ce6f85ab8 in term 1.
I20251211 08:58:28.977139 16916 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b; no voters:
I20251211 08:58:28.978147 16974 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:28.979672 16974 raft_consensus.cc:697] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 1 LEADER]: Becoming Leader. State: Replica: b8dd1b89d5ad4e529459606ce6f85ab8, State: Running, Role: LEADER
I20251211 08:58:28.980628 16974 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:28.999387 16999 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8dd1b89d5ad4e529459606ce6f85ab8. Latest consensus state: current_term: 1 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:29.000878 16999 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: LEADER
W20251211 08:58:29.011309 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:29.012265 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:29.029441 16988 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:29.030582 16988 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:58:29.031733 17005 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251211 08:58:29.038192 17005 catalog_manager.cc:1494] Initializing Kudu cluster ID...
W20251211 08:58:29.071084 17011 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:29.071468 17011 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:29.085600 17011 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:29.085991 17011 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251211 08:58:29.101053 17011 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:29.105146 17011 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:58:29.103812 16861 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Refusing update from remote peer b8dd1b89d5ad4e529459606ce6f85ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:58:29.106726 16999 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:29.055938 16813 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: OK, instance_id { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" instance_seqno: 1765443504563048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251211 08:58:29.119113 16813 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:58:29.150503 17013 mvcc.cc:204] Tried to move back new op lower bound from 7231256613251104768 to 7231256612808478720. Current Snapshot: MvccSnapshot[applied={T|T < 7231256613251104768}]
I20251211 08:58:29.155164 17005 catalog_manager.cc:1357] Generated new cluster ID: c258c1a306e24c4197d31ee43870f1fb
I20251211 08:58:29.155419 17005 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
W20251211 08:58:29.159917 17011 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20251211 08:58:29.161996 16993 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8dd1b89d5ad4e529459606ce6f85ab8. Latest consensus state: current_term: 1 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:29.163821 16974 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:29.164456 16974 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: LEADER
I20251211 08:58:29.162678 16993 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: FOLLOWER
W20251211 08:58:29.160192 17011 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:58:29.187436 16993 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:29.188104 16993 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: FOLLOWER
W20251211 08:58:29.190709 17011 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251211 08:58:29.191404 17011 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251211 08:58:29.201821 17005 catalog_manager.cc:1380] Generated new certificate authority record
I20251211 08:58:29.206328 17005 catalog_manager.cc:1514] Loading token signing keys...
I20251211 08:58:29.236364 17011 catalog_manager.cc:1269] Loaded cluster ID: c258c1a306e24c4197d31ee43870f1fb
I20251211 08:58:29.238019 17011 catalog_manager.cc:1562] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: loading cluster ID for follower catalog manager: success
I20251211 08:58:29.265141 17011 catalog_manager.cc:1584] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: acquiring CA information for follower catalog manager: success
W20251211 08:58:29.275374 17011 catalog_manager.cc:883] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20251211 08:58:29.282527 17005 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Generated new TSK 0
I20251211 08:58:29.283643 17005 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251211 08:58:29.287909 17005 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251211 08:58:29.294137 17011 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251211 08:58:29.298209 17005 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251211 08:58:29.300737 16812 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } attempt: 7
I20251211 08:58:29.324954 16812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: Bootstrap starting.
I20251211 08:58:29.344683 16812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:29.345916 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.146.1:0
--local_ip_for_outbound_sockets=127.13.146.1
--webserver_interface=127.13.146.1
--webserver_port=0
--tserver_master_addrs=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--builtin_ntp_servers=127.13.146.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251211 08:58:29.348652 16812 log.cc:826] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: Log is configured to *not* fsync() on all Append() calls
I20251211 08:58:29.363436 16812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: No bootstrap required, opened a new log
I20251211 08:58:29.437877 16812 raft_consensus.cc:359] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:29.439910 16812 raft_consensus.cc:385] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:29.440582 16812 raft_consensus.cc:740] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133269af8ce44b579ebbb7805bc58800, State: Initialized, Role: FOLLOWER
I20251211 08:58:29.443538 16812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
W20251211 08:58:29.452003 16916 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.13.146.62:43161 timed out after 0.509s (SENT)
I20251211 08:58:29.460397 17022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:29.461061 17022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:29.479496 16786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800"
I20251211 08:58:29.479846 16785 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:29.482970 16784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800" is_pre_election: true
I20251211 08:58:29.484162 16787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800" is_pre_election: true
W20251211 08:58:29.484935 16787 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44466 (request call id 2) took 542 ms (client timeout 509 ms). Trace:
W20251211 08:58:29.485272 16787 rpcz_store.cc:269] 1211 08:58:28.942513 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:28.942666 (+ 153us) service_pool.cc:225] Handling call
1211 08:58:29.484867 (+542201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251211 08:58:29.491638 16785 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Refusing update from remote peer b8dd1b89d5ad4e529459606ce6f85ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251211 08:58:29.494340 16999 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:29.504112 16812 sys_catalog.cc:565] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: configured and running, proceeding with master startup.
W20251211 08:58:29.529165 16766 tablet.cc:2378] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251211 08:58:29.612262 17029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:29.613055 17029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:29.613547 17022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8dd1b89d5ad4e529459606ce6f85ab8. Latest consensus state: current_term: 1 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:29.614178 17022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:29.695840 17044 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251211 08:58:29.735165 17044 catalog_manager.cc:1269] Loaded cluster ID: c258c1a306e24c4197d31ee43870f1fb
I20251211 08:58:29.735405 17044 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: loading cluster ID for follower catalog manager: success
I20251211 08:58:29.763121 17044 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: acquiring CA information for follower catalog manager: success
I20251211 08:58:29.782807 17044 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251211 08:58:29.946038 17046 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:29.950593 17046 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: LEADER
W20251211 08:58:30.048170 17021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:58:30.048761 17021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:58:30.049603 17021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:58:30.112789 17021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:58:30.116320 17021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.146.1
I20251211 08:58:30.183550 17021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.146.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.146.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.146.1
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:58:30.185314 17021 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:58:30.187628 17021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:58:30.231208 17055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:31.167528 17058 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:31.168485 17058 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:31.172011 16787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800" is_pre_election: true
I20251211 08:58:31.176859 17058 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:31.178092 16942 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:31.179698 16837 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:31.181644 17058 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:58:31.735607 17059 raft_consensus.cc:493] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:31.736617 17059 raft_consensus.cc:515] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:31.740819 16860 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b" is_pre_election: true
I20251211 08:58:31.749396 16942 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:31.750921 16763 leader_election.cc:304] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800; no voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:31.759696 17059 leader_election.cc:290] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:31.760679 17059 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:58:32.553551 17061 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:32.554243 17061 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:32.556814 16787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800" is_pre_election: true
I20251211 08:58:32.560352 16942 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:32.560891 17061 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:32.561762 16837 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:32.562458 17061 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251211 08:58:31.612663 17053 debug-util.cc:398] Leaking SignalData structure 0x7b0800034c40 after lost signal to thread 17021
W20251211 08:58:32.706842 17021 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 2.479s user 0.732s sys 1.245s
W20251211 08:58:32.707926 17021 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 2.480s user 0.732s sys 1.245s
W20251211 08:58:30.231698 17054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:32.722266 17064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:32.722478 17021 server_base.cc:1047] running on GCE node
I20251211 08:58:32.724064 17021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:58:32.726799 17021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:58:32.728224 17021 hybrid_clock.cc:648] HybridClock initialized: now 1765443512728122 us; error 94 us; skew 500 ppm
I20251211 08:58:32.729243 17021 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:58:32.737130 17021 webserver.cc:492] Webserver started at http://127.13.146.1:38695/ using document root <none> and password file <none>
I20251211 08:58:32.738549 17021 fs_manager.cc:362] Metadata directory not provided
I20251211 08:58:32.738880 17021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:58:32.739518 17021 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:58:32.745746 17021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/data/instance:
uuid: "62329f65aae54e84b396a51dd81f1b70"
format_stamp: "Formatted at 2025-12-11 08:58:32 on dist-test-slave-0dfs"
I20251211 08:58:32.747320 17021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/instance:
uuid: "62329f65aae54e84b396a51dd81f1b70"
format_stamp: "Formatted at 2025-12-11 08:58:32 on dist-test-slave-0dfs"
I20251211 08:58:32.756664 17021 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.003s
I20251211 08:58:32.765539 17069 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:32.766845 17021 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.004s
I20251211 08:58:32.767284 17021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal
uuid: "62329f65aae54e84b396a51dd81f1b70"
format_stamp: "Formatted at 2025-12-11 08:58:32 on dist-test-slave-0dfs"
I20251211 08:58:32.767812 17021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:58:32.858134 17074 raft_consensus.cc:493] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:32.858698 17074 raft_consensus.cc:515] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:32.860294 17021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:58:32.861354 16860 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b" is_pre_election: true
I20251211 08:58:32.862190 17021 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:58:32.862706 17021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:58:32.862109 17074 leader_election.cc:290] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:32.864028 16942 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:32.865309 16763 leader_election.cc:304] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800; no voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:32.866076 17021 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251211 08:58:32.866465 17074 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251211 08:58:32.871529 17021 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251211 08:58:32.871800 17021 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:32.872088 17021 ts_tablet_manager.cc:616] Registered 0 tablets
I20251211 08:58:32.872303 17021 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:33.046811 17021 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.1:46231
I20251211 08:58:33.046911 17182 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.1:46231 every 8 connection(s)
I20251211 08:58:33.049446 17021 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/data/info.pb
I20251211 08:58:33.060308 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 17021
I20251211 08:58:33.060837 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/instance
I20251211 08:58:33.082726 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.146.2:0
--local_ip_for_outbound_sockets=127.13.146.2
--webserver_interface=127.13.146.2
--webserver_port=0
--tserver_master_addrs=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--builtin_ntp_servers=127.13.146.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251211 08:58:33.098868 17183 heartbeater.cc:344] Connected to a master server at 127.13.146.60:34701
I20251211 08:58:33.099359 17183 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:33.100593 17183 heartbeater.cc:507] Master 127.13.146.60:34701 requested a full tablet report, sending...
I20251211 08:58:33.103499 16932 ts_manager.cc:194] Registered new tserver with Master: 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231)
I20251211 08:58:33.106195 16932 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.146.1:50379
I20251211 08:58:33.110735 17184 heartbeater.cc:344] Connected to a master server at 127.13.146.61:36035
I20251211 08:58:33.111086 17184 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:33.111792 17184 heartbeater.cc:507] Master 127.13.146.61:36035 requested a full tablet report, sending...
I20251211 08:58:33.112712 17185 heartbeater.cc:344] Connected to a master server at 127.13.146.62:43161
I20251211 08:58:33.113073 17185 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:33.113670 17185 heartbeater.cc:507] Master 127.13.146.62:43161 requested a full tablet report, sending...
I20251211 08:58:33.115669 16851 ts_manager.cc:194] Registered new tserver with Master: 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231)
I20251211 08:58:33.116055 16777 ts_manager.cc:194] Registered new tserver with Master: 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231)
W20251211 08:58:33.387276 17193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:58:33.387787 17193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:58:33.388302 17193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:58:33.418787 17193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:58:33.419976 17193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.146.2
I20251211 08:58:33.453963 17198 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:33.454308 17198 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:33.454727 17193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.146.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.146.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.146.2
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:58:33.455664 17198 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:33.456137 17193 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:58:33.456632 16787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800" is_pre_election: true
I20251211 08:58:33.456610 16942 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:33.457742 16837 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:33.457964 17193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251211 08:58:33.458287 17198 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251211 08:58:33.469532 17200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:33.487334 17202 raft_consensus.cc:493] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:33.487835 17202 raft_consensus.cc:515] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:33.491292 16942 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:33.494431 16860 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b" is_pre_election: true
I20251211 08:58:33.495193 16860 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133269af8ce44b579ebbb7805bc58800 in term 1.
I20251211 08:58:33.496903 16761 leader_election.cc:304] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800, c3112fc6f4ae4541a2af2405a97c707b; no voters: b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:33.497521 17202 leader_election.cc:290] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:33.498175 17202 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251211 08:58:33.498569 17202 raft_consensus.cc:493] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:33.499028 17202 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:58:33.527143 17202 raft_consensus.cc:515] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:33.530154 16860 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b"
I20251211 08:58:33.530645 16860 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:58:33.539402 16860 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133269af8ce44b579ebbb7805bc58800 in term 2.
I20251211 08:58:33.541538 16942 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8"
I20251211 08:58:33.543453 17202 leader_election.cc:290] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 2 election: Requested vote from peers c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:33.544574 16761 leader_election.cc:304] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800, c3112fc6f4ae4541a2af2405a97c707b; no voters: b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:33.545460 17208 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 2 FOLLOWER]: Leader election won for term 2
I20251211 08:58:33.545985 17208 raft_consensus.cc:697] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 2 LEADER]: Becoming Leader. State: Replica: 133269af8ce44b579ebbb7805bc58800, State: Running, Role: LEADER
I20251211 08:58:33.546785 17208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:33.552284 17208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133269af8ce44b579ebbb7805bc58800. Latest consensus state: current_term: 2 leader_uuid: "133269af8ce44b579ebbb7805bc58800" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:33.553364 17208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: This master's current role is: LEADER
I20251211 08:58:33.558303 16860 raft_consensus.cc:1240] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 2 FOLLOWER]: Rejecting Update request from peer b8dd1b89d5ad4e529459606ce6f85ab8 for earlier term 1. Current term is 2. Ops: []
I20251211 08:58:33.563186 17204 consensus_queue.cc:1059] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251211 08:58:33.567257 17211 raft_consensus.cc:3055] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 1 LEADER]: Stepping down as leader of term 1
I20251211 08:58:33.568547 17211 raft_consensus.cc:740] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b8dd1b89d5ad4e529459606ce6f85ab8, State: Running, Role: LEADER
I20251211 08:58:33.569846 17211 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:33.571784 17210 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251211 08:58:33.571830 17211 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 1 FOLLOWER]: Advancing to term 2
I20251211 08:58:33.592352 16988 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251211 08:58:33.607302 17210 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251211 08:58:33.610468 17183 heartbeater.cc:499] Master 127.13.146.60:34701 was elected leader, sending a full tablet report...
I20251211 08:58:33.612946 17210 catalog_manager.cc:1269] Loaded cluster ID: c258c1a306e24c4197d31ee43870f1fb
I20251211 08:58:33.613268 17210 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251211 08:58:33.631793 17210 catalog_manager.cc:1514] Loading token signing keys...
I20251211 08:58:33.653048 17210 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: Loaded TSK: 0
I20251211 08:58:33.654752 17210 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251211 08:58:33.671411 17210 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251211 08:58:33.684306 17210 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251211 08:58:34.072283 17211 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:34.072765 17211 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:34.074642 17211 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
I20251211 08:58:34.076792 16787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800" is_pre_election: true
I20251211 08:58:34.077066 16860 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b" is_pre_election: true
I20251211 08:58:34.077610 16860 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8dd1b89d5ad4e529459606ce6f85ab8 in term 2.
I20251211 08:58:34.079778 16916 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800
I20251211 08:58:34.081175 17211 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251211 08:58:34.081837 17211 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:34.082326 17211 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 2 FOLLOWER]: Advancing to term 3
I20251211 08:58:34.091684 17211 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:34.096247 16787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800"
I20251211 08:58:34.099712 16860 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b"
I20251211 08:58:34.100248 16860 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 2 FOLLOWER]: Advancing to term 3
I20251211 08:58:34.110904 17211 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 3 election: Requested vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
I20251211 08:58:34.110641 16860 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8dd1b89d5ad4e529459606ce6f85ab8 in term 3.
I20251211 08:58:34.112957 16916 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800
I20251211 08:58:34.114411 17211 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 3 FOLLOWER]: Leader election won for term 3
I20251211 08:58:34.120128 17211 raft_consensus.cc:697] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 3 LEADER]: Becoming Leader. State: Replica: b8dd1b89d5ad4e529459606ce6f85ab8, State: Running, Role: LEADER
I20251211 08:58:34.122926 17211 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:34.144186 16942 raft_consensus.cc:1240] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 3 LEADER]: Rejecting Update request from peer 133269af8ce44b579ebbb7805bc58800 for earlier term 2. Current term is 3. Ops: []
I20251211 08:58:34.149755 17212 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251211 08:58:34.151768 17213 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8dd1b89d5ad4e529459606ce6f85ab8. Latest consensus state: current_term: 3 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:34.152696 17213 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: LEADER
I20251211 08:58:34.154160 17215 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251211 08:58:34.156687 17212 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [LEADER]: Peer b8dd1b89d5ad4e529459606ce6f85ab8 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251211 08:58:34.157373 17216 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 2 LEADER]: Stepping down as leader of term 2
I20251211 08:58:34.158082 17216 raft_consensus.cc:740] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 133269af8ce44b579ebbb7805bc58800, State: Running, Role: LEADER
I20251211 08:58:34.159144 17216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:34.160367 17216 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 2 FOLLOWER]: Advancing to term 3
I20251211 08:58:34.170234 16813 master_runner.cc:232] Existing masters have differing terms: 2 vs 3
I20251211 08:58:34.170688 17215 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251211 08:58:34.170753 16813 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:58:34.173673 17215 catalog_manager.cc:1269] Loaded cluster ID: c258c1a306e24c4197d31ee43870f1fb
I20251211 08:58:34.174803 17215 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251211 08:58:34.182857 17215 catalog_manager.cc:1514] Loading token signing keys...
I20251211 08:58:34.193689 17215 catalog_manager.cc:6038] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Loaded TSK: 0
I20251211 08:58:34.195225 17215 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251211 08:58:34.201653 17215 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251211 08:58:34.209429 17215 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251211 08:58:34.555745 17216 raft_consensus.cc:493] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:34.556345 17216 raft_consensus.cc:515] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:34.561496 16942 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:34.564889 16860 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b" is_pre_election: true
I20251211 08:58:34.567533 16860 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133269af8ce44b579ebbb7805bc58800 in term 3.
I20251211 08:58:34.569072 16761 leader_election.cc:304] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800, c3112fc6f4ae4541a2af2405a97c707b; no voters: b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:34.570170 17212 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251211 08:58:34.570617 17212 raft_consensus.cc:493] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:34.571946 17212 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 3 FOLLOWER]: Advancing to term 4
I20251211 08:58:34.558606 17216 leader_election.cc:290] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:34.583283 16860 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 3 FOLLOWER]: Refusing update from remote peer b8dd1b89d5ad4e529459606ce6f85ab8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251211 08:58:34.585281 17213 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251211 08:58:34.598565 17212 raft_consensus.cc:515] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:34.610719 16861 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b"
I20251211 08:58:34.624316 16942 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8"
I20251211 08:58:34.626389 16763 leader_election.cc:304] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800; no voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:34.628039 17216 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251211 08:58:34.629165 17185 heartbeater.cc:499] Master 127.13.146.62:43161 was elected leader, sending a full tablet report...
I20251211 08:58:34.633415 17212 leader_election.cc:290] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 4 election: Requested vote from peers c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:34.657512 16787 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 4 FOLLOWER]: Rejecting Update request from peer b8dd1b89d5ad4e529459606ce6f85ab8 for earlier term 3. Current term is 4. Ops: []
I20251211 08:58:34.659036 17211 consensus_queue.cc:1059] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251211 08:58:34.659888 17211 consensus_queue.cc:1243] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Peer 133269af8ce44b579ebbb7805bc58800 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251211 08:58:34.662914 17222 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:34.663595 17222 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: LEADER
I20251211 08:58:34.664393 17213 raft_consensus.cc:3055] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 3 LEADER]: Stepping down as leader of term 3
I20251211 08:58:34.664829 17213 raft_consensus.cc:740] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b8dd1b89d5ad4e529459606ce6f85ab8, State: Running, Role: LEADER
I20251211 08:58:34.665894 17213 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:34.667064 17213 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 3 FOLLOWER]: Advancing to term 4
I20251211 08:58:34.674700 17217 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8dd1b89d5ad4e529459606ce6f85ab8. Latest consensus state: current_term: 3 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:34.675849 17217 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: FOLLOWER
W20251211 08:58:34.681079 17215 catalog_manager.cc:1419] Failed waiting for node to catch up after master election: Illegal state: Node b8dd1b89d5ad4e529459606ce6f85ab8 not leader. Raft Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:34.685137 17221 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:34.692159 17221 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:35.069001 17213 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:35.069470 17213 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:35.071573 17213 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
I20251211 08:58:35.075052 16787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800" is_pre_election: true
I20251211 08:58:35.075590 16787 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8dd1b89d5ad4e529459606ce6f85ab8 in term 4.
I20251211 08:58:35.077458 16916 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8; no voters:
I20251211 08:58:35.078400 17213 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251211 08:58:35.078770 17213 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:35.077581 16861 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b" is_pre_election: true
I20251211 08:58:35.079205 17213 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 4 FOLLOWER]: Advancing to term 5
I20251211 08:58:35.089671 17213 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:35.093590 16787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800"
I20251211 08:58:35.094190 16787 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 4 FOLLOWER]: Advancing to term 5
W20251211 08:58:34.872687 17199 debug-util.cc:398] Leaking SignalData structure 0x7b0800034c40 after lost signal to thread 17193
W20251211 08:58:35.096714 17199 kernel_stack_watchdog.cc:198] Thread 17193 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20251211 08:58:33.471504 17201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:35.101037 17193 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.630s user 0.476s sys 1.001s
W20251211 08:58:35.101454 17193 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.630s user 0.476s sys 1.001s
I20251211 08:58:35.101974 17213 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 5 election: Requested vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
I20251211 08:58:35.103063 16861 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b"
I20251211 08:58:35.103327 16787 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8dd1b89d5ad4e529459606ce6f85ab8 in term 5.
W20251211 08:58:35.104024 17205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:35.104297 17203 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1631 milliseconds
I20251211 08:58:35.104882 17193 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251211 08:58:35.104987 16916 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8; no voters: c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:35.105523 17213 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 5 FOLLOWER]: Leader election won for term 5
I20251211 08:58:35.105978 17213 raft_consensus.cc:697] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 5 LEADER]: Becoming Leader. State: Replica: b8dd1b89d5ad4e529459606ce6f85ab8, State: Running, Role: LEADER
I20251211 08:58:35.106103 17193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:58:35.106673 17213 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:35.108462 17193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:58:35.109508 17211 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8dd1b89d5ad4e529459606ce6f85ab8. Latest consensus state: current_term: 5 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:35.109972 17211 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: LEADER
I20251211 08:58:35.109992 17193 hybrid_clock.cc:648] HybridClock initialized: now 1765443515109923 us; error 58 us; skew 500 ppm
I20251211 08:58:35.110536 17215 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251211 08:58:35.110936 17193 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:58:35.115974 17215 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251211 08:58:35.117285 17193 webserver.cc:492] Webserver started at http://127.13.146.2:36175/ using document root <none> and password file <none>
I20251211 08:58:35.118173 17193 fs_manager.cc:362] Metadata directory not provided
I20251211 08:58:35.118366 17193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:58:35.118825 17193 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:58:35.118873 17215 catalog_manager.cc:1269] Loaded cluster ID: c258c1a306e24c4197d31ee43870f1fb
I20251211 08:58:35.119100 17215 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251211 08:58:35.121909 17215 catalog_manager.cc:1514] Loading token signing keys...
I20251211 08:58:35.123306 17193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/data/instance:
uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2"
format_stamp: "Formatted at 2025-12-11 08:58:35 on dist-test-slave-0dfs"
I20251211 08:58:35.124516 17215 catalog_manager.cc:6038] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Loaded TSK: 0
I20251211 08:58:35.124742 17193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/instance:
uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2"
format_stamp: "Formatted at 2025-12-11 08:58:35 on dist-test-slave-0dfs"
I20251211 08:58:35.125218 17215 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251211 08:58:35.128212 17215 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251211 08:58:35.130730 17215 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251211 08:58:35.132447 17193 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20251211 08:58:35.138602 17228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:35.139729 17193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20251211 08:58:35.140101 17193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal
uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2"
format_stamp: "Formatted at 2025-12-11 08:58:35 on dist-test-slave-0dfs"
I20251211 08:58:35.140471 17193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:58:35.147157 17221 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:35.147630 17221 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:35.149580 17221 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:35.150287 16787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800" is_pre_election: true
I20251211 08:58:35.150967 16787 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3112fc6f4ae4541a2af2405a97c707b for earlier term 4. Current term is 5.
I20251211 08:58:35.150796 16942 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:35.152285 16835 leader_election.cc:400] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 4 pre-election: Vote denied by peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3112fc6f4ae4541a2af2405a97c707b for earlier term 4. Current term is 5.
I20251211 08:58:35.152494 16835 leader_election.cc:403] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251211 08:58:35.153146 17221 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 3 FOLLOWER]: Advancing to term 5
I20251211 08:58:35.158839 17221 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3112fc6f4ae4541a2af2405a97c707b for earlier term 4. Current term is 5.
I20251211 08:58:35.165069 17183 heartbeater.cc:499] Master 127.13.146.60:34701 was elected leader, sending a full tablet report...
I20251211 08:58:35.192647 17193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:58:35.193965 17193 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:58:35.194351 17193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:58:35.196851 17193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251211 08:58:35.201272 17193 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251211 08:58:35.201500 17193 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:35.201726 17193 ts_tablet_manager.cc:616] Registered 0 tablets
I20251211 08:58:35.201874 17193 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:35.334637 17193 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.2:40951
I20251211 08:58:35.334714 17340 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.2:40951 every 8 connection(s)
I20251211 08:58:35.336977 17193 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/data/info.pb
I20251211 08:58:35.344209 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 17193
I20251211 08:58:35.344830 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/instance
I20251211 08:58:35.361733 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.146.3:0
--local_ip_for_outbound_sockets=127.13.146.3
--webserver_interface=127.13.146.3
--webserver_port=0
--tserver_master_addrs=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--builtin_ntp_servers=127.13.146.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251211 08:58:35.381475 17342 heartbeater.cc:344] Connected to a master server at 127.13.146.61:36035
I20251211 08:58:35.381951 17342 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:35.383198 17342 heartbeater.cc:507] Master 127.13.146.61:36035 requested a full tablet report, sending...
I20251211 08:58:35.386693 16851 ts_manager.cc:194] Registered new tserver with Master: e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
I20251211 08:58:35.388247 17341 heartbeater.cc:344] Connected to a master server at 127.13.146.60:34701
I20251211 08:58:35.388598 17341 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:35.389333 17341 heartbeater.cc:507] Master 127.13.146.60:34701 requested a full tablet report, sending...
I20251211 08:58:35.391324 17343 heartbeater.cc:344] Connected to a master server at 127.13.146.62:43161
I20251211 08:58:35.391417 16932 ts_manager.cc:194] Registered new tserver with Master: e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
I20251211 08:58:35.391685 17343 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:35.392549 17343 heartbeater.cc:507] Master 127.13.146.62:43161 requested a full tablet report, sending...
I20251211 08:58:35.393548 16932 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.146.2:44759
I20251211 08:58:35.395216 16777 ts_manager.cc:194] Registered new tserver with Master: e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 08:58:35.696910 17348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:58:35.697346 17348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:58:35.697860 17348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251211 08:58:35.710132 16787 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 5 FOLLOWER]: Refusing update from remote peer b8dd1b89d5ad4e529459606ce6f85ab8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251211 08:58:35.711577 17356 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20251211 08:58:35.711985 17356 consensus_queue.cc:1243] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Peer 133269af8ce44b579ebbb7805bc58800 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251211 08:58:35.713971 16787 pending_rounds.cc:85] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: Aborting all ops after (but not including) 4
I20251211 08:58:35.714269 16787 pending_rounds.cc:107] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20251211 08:58:35.714538 16787 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251211 08:58:35.719422 16861 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 5 FOLLOWER]: Refusing update from remote peer b8dd1b89d5ad4e529459606ce6f85ab8: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251211 08:58:35.721086 17357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8dd1b89d5ad4e529459606ce6f85ab8. Latest consensus state: current_term: 5 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:35.721951 17357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:35.723929 17362 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251211 08:58:35.720081 17360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:35.725662 17360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:35.728775 17360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:35.729373 17360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:35.737628 17361 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:35.738281 17361 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: LEADER
W20251211 08:58:35.742758 17348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:58:35.743894 17348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.146.3
I20251211 08:58:35.748350 17368 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:35.749148 17368 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:35.749034 17364 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8dd1b89d5ad4e529459606ce6f85ab8. Latest consensus state: current_term: 5 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:35.749716 17364 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:35.779611 17348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.146.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.146.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.146.3
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:58:35.780853 17348 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:58:35.782420 17348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:58:35.793610 17370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:35.896765 17341 heartbeater.cc:499] Master 127.13.146.60:34701 was elected leader, sending a full tablet report...
I20251211 08:58:36.161249 17362 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:36.161868 17362 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: LEADER
W20251211 08:58:35.798251 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:35.795049 17371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:36.977073 17372 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20251211 08:58:36.977229 17348 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20251211 08:58:36.982014 17348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:58:36.984479 17348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:58:36.985797 17348 hybrid_clock.cc:648] HybridClock initialized: now 1765443516985770 us; error 40 us; skew 500 ppm
I20251211 08:58:36.986562 17348 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:58:36.992596 17348 webserver.cc:492] Webserver started at http://127.13.146.3:33981/ using document root <none> and password file <none>
I20251211 08:58:36.993444 17348 fs_manager.cc:362] Metadata directory not provided
I20251211 08:58:36.993623 17348 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:58:36.994012 17348 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251211 08:58:36.998296 17348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/data/instance:
uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf"
format_stamp: "Formatted at 2025-12-11 08:58:36 on dist-test-slave-0dfs"
I20251211 08:58:36.999557 17348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal/instance:
uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf"
format_stamp: "Formatted at 2025-12-11 08:58:36 on dist-test-slave-0dfs"
I20251211 08:58:37.006425 17348 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20251211 08:58:37.011873 17381 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:37.012810 17348 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20251211 08:58:37.013125 17348 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal
uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf"
format_stamp: "Formatted at 2025-12-11 08:58:36 on dist-test-slave-0dfs"
I20251211 08:58:37.013422 17348 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:58:37.073109 17348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:58:37.074534 17348 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:58:37.074965 17348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:58:37.077709 17348 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251211 08:58:37.082159 17348 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251211 08:58:37.082355 17348 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:37.082553 17348 ts_tablet_manager.cc:616] Registered 0 tablets
I20251211 08:58:37.082719 17348 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:37.221690 17348 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.3:41219
I20251211 08:58:37.221793 17493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.3:41219 every 8 connection(s)
I20251211 08:58:37.224221 17348 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/data/info.pb
I20251211 08:58:37.229473 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 17348
I20251211 08:58:37.230010 13896 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal/instance
I20251211 08:58:37.267812 17495 heartbeater.cc:344] Connected to a master server at 127.13.146.61:36035
I20251211 08:58:37.268270 17495 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:37.269440 17495 heartbeater.cc:507] Master 127.13.146.61:36035 requested a full tablet report, sending...
I20251211 08:58:37.270455 17496 heartbeater.cc:344] Connected to a master server at 127.13.146.62:43161
I20251211 08:58:37.270854 17496 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:37.271087 17494 heartbeater.cc:344] Connected to a master server at 127.13.146.60:34701
I20251211 08:58:37.271411 17494 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:37.271661 17496 heartbeater.cc:507] Master 127.13.146.62:43161 requested a full tablet report, sending...
I20251211 08:58:37.272202 17494 heartbeater.cc:507] Master 127.13.146.60:34701 requested a full tablet report, sending...
I20251211 08:58:37.272769 16851 ts_manager.cc:194] Registered new tserver with Master: 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:37.273866 16777 ts_manager.cc:194] Registered new tserver with Master: 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:37.274187 16932 ts_manager.cc:194] Registered new tserver with Master: 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:37.275904 16932 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.146.3:35161
I20251211 08:58:37.285979 13896 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251211 08:58:37.327610 16932 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33244:
name: "default.Table_1bf93acd2af64f98b4cda7ce375e702e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:37.328939 16931 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33244:
name: "default.Table_da60da9af7664a10a99d2fa227af8235"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:37.338218 16932 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1bf93acd2af64f98b4cda7ce375e702e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:37.339748 16930 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33244:
name: "default.Table_d36118f54fec423badd3d2b59d64b05c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:37.342523 16929 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33244:
name: "default.Table_12ea97bf52084b09955460036752d6e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:37.346266 16931 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_da60da9af7664a10a99d2fa227af8235 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:37.375020 13896 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu with pid 16887
W20251211 08:58:37.407141 17506 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.146.60:34701)
W20251211 08:58:37.415474 17231 connection.cc:537] client connection to 127.13.146.60:34701 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:37.416158 17341 heartbeater.cc:646] Failed to heartbeat to 127.13.146.60:34701 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:37.425354 17510 negotiation.cc:337] Failed RPC negotiation. Trace:
1211 08:58:37.421931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.146.60:34701 (local address 127.0.0.1:33258)
1211 08:58:37.423561 (+ 1630us) negotiation.cc:107] Waiting for socket to connect
1211 08:58:37.423911 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":676}
I20251211 08:58:37.540086 17529 raft_consensus.cc:493] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:37.540683 17529 raft_consensus.cc:515] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:37.543995 16861 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b" is_pre_election: true
I20251211 08:58:37.544104 17529 leader_election.cc:290] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:58:37.565335 16763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20251211 08:58:37.567499 17531 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:37.568344 17531 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
W20251211 08:58:37.570149 16763 leader_election.cc:336] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:37.570504 16763 leader_election.cc:304] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800; no voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:37.570760 16787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800" is_pre_election: true
I20251211 08:58:37.571570 16787 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3112fc6f4ae4541a2af2405a97c707b in term 5.
I20251211 08:58:37.572340 17529 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251211 08:58:37.573277 16835 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800, c3112fc6f4ae4541a2af2405a97c707b; no voters:
I20251211 08:58:37.574414 17531 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:37.574942 17531 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251211 08:58:37.575065 16837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20251211 08:58:37.575332 17531 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 5 FOLLOWER]: Starting leader election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:37.575742 17531 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 5 FOLLOWER]: Advancing to term 6
W20251211 08:58:37.580220 16837 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:37.588163 17531 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:37.591017 16787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "133269af8ce44b579ebbb7805bc58800"
I20251211 08:58:37.591533 16787 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 5 FOLLOWER]: Advancing to term 6
I20251211 08:58:37.592427 17531 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 6 election: Requested vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:58:37.594115 16837 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:37.597949 16787 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3112fc6f4ae4541a2af2405a97c707b in term 6.
I20251211 08:58:37.599987 16835 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800, c3112fc6f4ae4541a2af2405a97c707b; no voters: b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:37.601856 17531 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 6 FOLLOWER]: Leader election won for term 6
I20251211 08:58:37.602418 17531 raft_consensus.cc:697] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 6 LEADER]: Becoming Leader. State: Replica: c3112fc6f4ae4541a2af2405a97c707b, State: Running, Role: LEADER
I20251211 08:58:37.603438 17531 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:37.615867 17534 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3112fc6f4ae4541a2af2405a97c707b. Latest consensus state: current_term: 6 leader_uuid: "c3112fc6f4ae4541a2af2405a97c707b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:37.616601 17534 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: LEADER
I20251211 08:58:37.627547 17537 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251211 08:58:37.637241 17537 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251211 08:58:37.641170 17537 catalog_manager.cc:1269] Loaded cluster ID: c258c1a306e24c4197d31ee43870f1fb
I20251211 08:58:37.642128 17537 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251211 08:58:37.646343 17537 catalog_manager.cc:1514] Loading token signing keys...
I20251211 08:58:37.652211 17537 catalog_manager.cc:6038] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Loaded TSK: 0
I20251211 08:58:37.655726 17537 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251211 08:58:37.663743 17537 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251211 08:58:37.666961 17537 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251211 08:58:37.694136 16851 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39386:
name: "default.Table_12ea97bf52084b09955460036752d6e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:37.699227 17183 heartbeater.cc:646] Failed to heartbeat to 127.13.146.60:34701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:37.702422 16850 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39386:
name: "default.Table_d36118f54fec423badd3d2b59d64b05c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:37.702952 16848 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39386:
name: "default.Table_1bf93acd2af64f98b4cda7ce375e702e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:37.704456 16849 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39386:
name: "default.Table_da60da9af7664a10a99d2fa227af8235"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251211 08:58:37.709005 16851 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_12ea97bf52084b09955460036752d6e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:58:37.712467 16849 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_da60da9af7664a10a99d2fa227af8235 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:58:37.714745 16848 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1bf93acd2af64f98b4cda7ce375e702e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251211 08:58:37.712519 16850 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d36118f54fec423badd3d2b59d64b05c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251211 08:58:37.779237 17494 heartbeater.cc:499] Master 127.13.146.60:34701 was elected leader, sending a full tablet report...
W20251211 08:58:37.788754 17494 heartbeater.cc:646] Failed to heartbeat to 127.13.146.60:34701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:37.926782 16850 catalog_manager.cc:2174] created HMS catalog entry for table [id=53d392cd36244b498c2740a372a1daf4]
W20251211 08:58:37.949259 16837 consensus_peers.cc:597] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b -> Peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Couldn't send request to peer b8dd1b89d5ad4e529459606ce6f85ab8. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:37.949331 16787 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Refusing update from remote peer c3112fc6f4ae4541a2af2405a97c707b: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251211 08:58:37.950623 17534 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [LEADER]: Connected to new peer: Peer: permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251211 08:58:37.964489 16849 catalog_manager.cc:2174] created HMS catalog entry for table [id=0869b45d01994929ba2c3284c458c299]
I20251211 08:58:37.966547 17529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3112fc6f4ae4541a2af2405a97c707b. Latest consensus state: current_term: 6 leader_uuid: "c3112fc6f4ae4541a2af2405a97c707b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:37.967212 17529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:37.972502 17541 mvcc.cc:204] Tried to move back new op lower bound from 7231256649496023040 to 7231256648114573312. Current Snapshot: MvccSnapshot[applied={T|T < 7231256649496023040}]
I20251211 08:58:37.987941 17531 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "c3112fc6f4ae4541a2af2405a97c707b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:37.988613 17531 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: LEADER
I20251211 08:58:37.995293 17529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "c3112fc6f4ae4541a2af2405a97c707b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:37.995919 17529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:38.003610 16850 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39386:
table { table_name: "default.Table_d36118f54fec423badd3d2b59d64b05c" } new_table_name: "default.Table_c00e0c8e65c24aac8676a17546efe6eb" modify_external_catalogs: true
I20251211 08:58:38.006366 16849 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39386:
table { table_name: "default.Table_da60da9af7664a10a99d2fa227af8235" } new_table_name: "default.Table_59e9546b87cc408f961a8f029510dc53" modify_external_catalogs: true
I20251211 08:58:38.071456 17429 tablet_service.cc:1505] Processing CreateTablet for tablet e6e62fc302554a84bcb15f0c8f97cba5 (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.073624 17428 tablet_service.cc:1505] Processing CreateTablet for tablet 0a6e1fd3f0e04978b128695735e5d038 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.075079 17428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a6e1fd3f0e04978b128695735e5d038. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.077412 17429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6e62fc302554a84bcb15f0c8f97cba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.080593 17276 tablet_service.cc:1505] Processing CreateTablet for tablet e6e62fc302554a84bcb15f0c8f97cba5 (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.082482 17276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6e62fc302554a84bcb15f0c8f97cba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.092546 17275 tablet_service.cc:1505] Processing CreateTablet for tablet 0a6e1fd3f0e04978b128695735e5d038 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.093664 17275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a6e1fd3f0e04978b128695735e5d038. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.097468 17117 tablet_service.cc:1505] Processing CreateTablet for tablet 0a6e1fd3f0e04978b128695735e5d038 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.099390 17118 tablet_service.cc:1505] Processing CreateTablet for tablet e6e62fc302554a84bcb15f0c8f97cba5 (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.105746 17117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a6e1fd3f0e04978b128695735e5d038. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.107883 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.60:34701
--webserver_interface=127.13.146.60
--webserver_port=41887
--builtin_ntp_servers=127.13.146.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--hive_metastore_uris=thrift://127.0.0.1:46123
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251211 08:58:38.111478 17118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6e62fc302554a84bcb15f0c8f97cba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.155607 17555 tablet_bootstrap.cc:492] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
I20251211 08:58:38.159351 17553 tablet_bootstrap.cc:492] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:38.167220 17553 tablet_bootstrap.cc:654] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.170926 17553 log.cc:826] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Log is configured to *not* fsync() on all Append() calls
I20251211 08:58:38.173130 17555 tablet_bootstrap.cc:654] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.175808 17555 log.cc:826] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Log is configured to *not* fsync() on all Append() calls
I20251211 08:58:38.183755 17553 tablet_bootstrap.cc:492] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:38.184242 17553 ts_tablet_manager.cc:1403] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.009s
I20251211 08:58:38.194880 17555 tablet_bootstrap.cc:492] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
I20251211 08:58:38.195375 17555 ts_tablet_manager.cc:1403] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.000s
I20251211 08:58:38.198417 17558 tablet_bootstrap.cc:492] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
I20251211 08:58:38.202121 16847 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39386:
name: "default.Table_eff061b04cb14ce88f2a8555db7504ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:38.207510 16846 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39386:
name: "default.Table_097cc85012564d15aa8db749266d3068"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:38.220721 16845 master_service.cc:954] ReplaceTablet: received request to replace tablet e6e62fc302554a84bcb15f0c8f97cba5 from {username='slave'} at 127.0.0.1:39386
I20251211 08:58:38.223060 17558 tablet_bootstrap.cc:654] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.226742 16844 master_service.cc:954] ReplaceTablet: received request to replace tablet 0a6e1fd3f0e04978b128695735e5d038 from {username='slave'} at 127.0.0.1:39386
I20251211 08:58:38.232849 17558 log.cc:826] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70: Log is configured to *not* fsync() on all Append() calls
I20251211 08:58:38.241284 17555 raft_consensus.cc:359] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:38.242362 17555 raft_consensus.cc:385] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.242756 17555 raft_consensus.cc:740] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.240939 17553 raft_consensus.cc:359] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.245549 17553 raft_consensus.cc:385] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.245832 17553 raft_consensus.cc:740] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.247056 17555 consensus_queue.cc:260] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:38.248734 17558 tablet_bootstrap.cc:492] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:38.249264 17558 ts_tablet_manager.cc:1403] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.051s user 0.017s sys 0.000s
I20251211 08:58:38.254659 17553 consensus_queue.cc:260] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.264322 16844 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Sending DeleteTablet for 3 replicas of tablet 0a6e1fd3f0e04978b128695735e5d038
I20251211 08:58:38.268819 17342 heartbeater.cc:499] Master 127.13.146.61:36035 was elected leader, sending a full tablet report...
I20251211 08:58:38.270257 17553 ts_tablet_manager.cc:1434] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.086s user 0.030s sys 0.006s
I20251211 08:58:38.273947 17555 ts_tablet_manager.cc:1434] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.078s user 0.021s sys 0.015s
I20251211 08:58:38.274781 16844 catalog_manager.cc:6724] ReplaceTablet: tablet 0a6e1fd3f0e04978b128695735e5d038 deleted and replaced by tablet a697ef839f1045a39f8621e5cd9f7233
I20251211 08:58:38.275020 17495 heartbeater.cc:499] Master 127.13.146.61:36035 was elected leader, sending a full tablet report...
I20251211 08:58:38.277189 17117 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a6e1fd3f0e04978b128695735e5d038 with delete_type TABLET_DATA_DELETED (replaced by tablet a697ef839f1045a39f8621e5cd9f7233) from {username='slave'} at 127.0.0.1:43290
I20251211 08:58:38.277504 17429 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a6e1fd3f0e04978b128695735e5d038 with delete_type TABLET_DATA_DELETED (replaced by tablet a697ef839f1045a39f8621e5cd9f7233) from {username='slave'} at 127.0.0.1:52092
I20251211 08:58:38.277693 16845 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Sending DeleteTablet for 3 replicas of tablet e6e62fc302554a84bcb15f0c8f97cba5
I20251211 08:58:38.278923 17553 tablet_bootstrap.cc:492] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:38.279178 17275 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a6e1fd3f0e04978b128695735e5d038 with delete_type TABLET_DATA_DELETED (replaced by tablet a697ef839f1045a39f8621e5cd9f7233) from {username='slave'} at 127.0.0.1:37278
I20251211 08:58:38.279793 17118 tablet_service.cc:1552] Processing DeleteTablet for tablet e6e62fc302554a84bcb15f0c8f97cba5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bb2fe37aa9741a9b892cf3103b1b6de) from {username='slave'} at 127.0.0.1:43290
I20251211 08:58:38.285347 17428 tablet_service.cc:1552] Processing DeleteTablet for tablet e6e62fc302554a84bcb15f0c8f97cba5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bb2fe37aa9741a9b892cf3103b1b6de) from {username='slave'} at 127.0.0.1:52092
I20251211 08:58:38.285898 17553 tablet_bootstrap.cc:654] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.286667 16845 catalog_manager.cc:6724] ReplaceTablet: tablet e6e62fc302554a84bcb15f0c8f97cba5 deleted and replaced by tablet 0bb2fe37aa9741a9b892cf3103b1b6de
I20251211 08:58:38.288880 17276 tablet_service.cc:1552] Processing DeleteTablet for tablet e6e62fc302554a84bcb15f0c8f97cba5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bb2fe37aa9741a9b892cf3103b1b6de) from {username='slave'} at 127.0.0.1:37278
I20251211 08:58:38.296393 17555 tablet_bootstrap.cc:492] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
I20251211 08:58:38.298534 17564 tablet_replica.cc:333] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: stopping tablet replica
W20251211 08:58:38.298389 16836 catalog_manager.cc:4965] TS 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): delete failed for tablet 0a6e1fd3f0e04978b128695735e5d038 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a6e1fd3f0e04978b128695735e5d038 already in progress: creating tablet
I20251211 08:58:38.299481 17564 raft_consensus.cc:2243] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:38.305320 17564 raft_consensus.cc:2272] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:38.308625 17564 ts_tablet_manager.cc:1916] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:38.309087 17555 tablet_bootstrap.cc:654] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.315900 17565 tablet_replica.cc:333] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2: stopping tablet replica
I20251211 08:58:38.316758 17565 raft_consensus.cc:2243] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:38.317308 17565 raft_consensus.cc:2272] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:38.318089 17429 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a6e1fd3f0e04978b128695735e5d038 with delete_type TABLET_DATA_DELETED (replaced by tablet a697ef839f1045a39f8621e5cd9f7233) from {username='slave'} at 127.0.0.1:52092
I20251211 08:58:38.323990 17565 ts_tablet_manager.cc:1916] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251211 08:58:38.327344 16835 catalog_manager.cc:4965] TS 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): delete failed for tablet 0a6e1fd3f0e04978b128695735e5d038 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a6e1fd3f0e04978b128695735e5d038 already in progress: creating tablet
W20251211 08:58:38.328027 16835 catalog_manager.cc:4965] TS 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): delete failed for tablet e6e62fc302554a84bcb15f0c8f97cba5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6e62fc302554a84bcb15f0c8f97cba5 already in progress: creating tablet
I20251211 08:58:38.334270 17553 tablet_bootstrap.cc:492] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:38.334678 17553 ts_tablet_manager.cc:1403] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.000s
I20251211 08:58:38.337230 17553 raft_consensus.cc:359] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:38.337893 17553 raft_consensus.cc:385] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.338173 17553 raft_consensus.cc:740] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.338840 17553 consensus_queue.cc:260] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:38.340878 17553 ts_tablet_manager.cc:1434] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251211 08:58:38.345472 17429 tablet_service.cc:1505] Processing CreateTablet for tablet 0bb2fe37aa9741a9b892cf3103b1b6de (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.346552 17429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bb2fe37aa9741a9b892cf3103b1b6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.350148 17555 tablet_bootstrap.cc:492] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
I20251211 08:58:38.350579 17555 ts_tablet_manager.cc:1403] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.004s
I20251211 08:58:38.353174 17555 raft_consensus.cc:359] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.353823 17555 raft_consensus.cc:385] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.354166 17555 raft_consensus.cc:740] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.354844 17555 consensus_queue.cc:260] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.361997 17555 ts_tablet_manager.cc:1434] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20251211 08:58:38.368634 17117 tablet_service.cc:1505] Processing CreateTablet for tablet 0bb2fe37aa9741a9b892cf3103b1b6de (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.369834 17117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bb2fe37aa9741a9b892cf3103b1b6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.373557 17275 tablet_service.cc:1505] Processing CreateTablet for tablet 0bb2fe37aa9741a9b892cf3103b1b6de (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.374490 17275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bb2fe37aa9741a9b892cf3103b1b6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.378532 17564 ts_tablet_manager.cc:1929] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:38.379168 17564 log.cc:1199] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/e6e62fc302554a84bcb15f0c8f97cba5
I20251211 08:58:38.380146 17564 ts_tablet_manager.cc:1950] T e6e62fc302554a84bcb15f0c8f97cba5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting consensus metadata
W20251211 08:58:38.381732 17349 tablet.cc:2378] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251211 08:58:38.390833 17564 tablet_replica.cc:333] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf: stopping tablet replica
I20251211 08:58:38.391135 16836 catalog_manager.cc:4985] TS 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): tablet e6e62fc302554a84bcb15f0c8f97cba5 (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]) successfully deleted
I20251211 08:58:38.395977 17555 tablet_bootstrap.cc:492] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
I20251211 08:58:38.400607 17558 raft_consensus.cc:359] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.401533 17558 raft_consensus.cc:385] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.401788 17558 raft_consensus.cc:740] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.402650 17558 consensus_queue.cc:260] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.404954 17555 tablet_bootstrap.cc:654] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.408609 17564 raft_consensus.cc:2243] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:38.411463 17564 raft_consensus.cc:2272] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:38.416417 17429 tablet_service.cc:1505] Processing CreateTablet for tablet a697ef839f1045a39f8621e5cd9f7233 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.414556 17564 ts_tablet_manager.cc:1916] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:38.417884 17429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a697ef839f1045a39f8621e5cd9f7233. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.419955 17553 tablet_bootstrap.cc:492] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:38.421650 17565 ts_tablet_manager.cc:1929] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:38.425689 17553 tablet_bootstrap.cc:654] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.428243 17565 log.cc:1199] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/0a6e1fd3f0e04978b128695735e5d038
I20251211 08:58:38.429512 17565 ts_tablet_manager.cc:1950] T 0a6e1fd3f0e04978b128695735e5d038 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting consensus metadata
I20251211 08:58:38.437155 16835 catalog_manager.cc:4985] TS e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): tablet 0a6e1fd3f0e04978b128695735e5d038 (table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]) successfully deleted
I20251211 08:58:38.443473 17117 tablet_service.cc:1505] Processing CreateTablet for tablet a697ef839f1045a39f8621e5cd9f7233 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.444623 17117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a697ef839f1045a39f8621e5cd9f7233. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.450665 17555 tablet_bootstrap.cc:492] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
I20251211 08:58:38.451161 17555 ts_tablet_manager.cc:1403] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.006s
I20251211 08:58:38.456079 17275 tablet_service.cc:1505] Processing CreateTablet for tablet a697ef839f1045a39f8621e5cd9f7233 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.457043 17275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a697ef839f1045a39f8621e5cd9f7233. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.464306 17553 tablet_bootstrap.cc:492] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:38.465102 17565 tablet_replica.cc:333] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: stopping tablet replica
I20251211 08:58:38.468032 17184 heartbeater.cc:499] Master 127.13.146.61:36035 was elected leader, sending a full tablet report...
I20251211 08:58:38.468881 17558 ts_tablet_manager.cc:1434] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.219s user 0.038s sys 0.026s
I20251211 08:58:38.471786 17565 raft_consensus.cc:2243] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:38.472277 17565 raft_consensus.cc:2272] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:38.465446 17553 ts_tablet_manager.cc:1403] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.018s
I20251211 08:58:38.454221 17555 raft_consensus.cc:359] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.474433 17555 raft_consensus.cc:385] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.474722 17555 raft_consensus.cc:740] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.475533 17555 consensus_queue.cc:260] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.477456 17555 ts_tablet_manager.cc:1434] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.026s user 0.004s sys 0.000s
I20251211 08:58:38.483445 16843 master_service.cc:954] ReplaceTablet: received request to replace tablet e6e62fc302554a84bcb15f0c8f97cba5 from {username='slave'} at 127.0.0.1:39386
I20251211 08:58:38.485932 17558 tablet_bootstrap.cc:492] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
I20251211 08:58:38.486706 17118 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a6e1fd3f0e04978b128695735e5d038 with delete_type TABLET_DATA_DELETED (replaced by tablet a697ef839f1045a39f8621e5cd9f7233) from {username='slave'} at 127.0.0.1:43290
I20251211 08:58:38.487735 17563 tablet_replica.cc:333] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70: stopping tablet replica
I20251211 08:58:38.488520 17563 raft_consensus.cc:2243] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:38.488590 17118 tablet_service.cc:1552] Processing DeleteTablet for tablet e6e62fc302554a84bcb15f0c8f97cba5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bb2fe37aa9741a9b892cf3103b1b6de) from {username='slave'} at 127.0.0.1:43290
I20251211 08:58:38.489113 17563 raft_consensus.cc:2272] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:38.491641 17558 tablet_bootstrap.cc:654] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.494105 17563 ts_tablet_manager.cc:1916] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:38.494343 16843 master_service.cc:954] ReplaceTablet: received request to replace tablet 0bb2fe37aa9741a9b892cf3103b1b6de from {username='slave'} at 127.0.0.1:39386
W20251211 08:58:38.503337 17501 tablet.cc:2378] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251211 08:58:38.482632 17553 raft_consensus.cc:359] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.504158 17553 raft_consensus.cc:385] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.514425 16843 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Sending DeleteTablet for 3 replicas of tablet 0bb2fe37aa9741a9b892cf3103b1b6de
I20251211 08:58:38.515664 17558 tablet_bootstrap.cc:492] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:38.516045 17558 ts_tablet_manager.cc:1403] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.001s
I20251211 08:58:38.516659 17429 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bb2fe37aa9741a9b892cf3103b1b6de with delete_type TABLET_DATA_DELETED (replaced by tablet 0963f28287b3483daca2b517f2b2c7f5) from {username='slave'} at 127.0.0.1:52092
I20251211 08:58:38.517680 17117 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bb2fe37aa9741a9b892cf3103b1b6de with delete_type TABLET_DATA_DELETED (replaced by tablet 0963f28287b3483daca2b517f2b2c7f5) from {username='slave'} at 127.0.0.1:43290
I20251211 08:58:38.519395 17558 raft_consensus.cc:359] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:38.520044 17558 raft_consensus.cc:385] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.520313 17558 raft_consensus.cc:740] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.520850 16843 catalog_manager.cc:6724] ReplaceTablet: tablet 0bb2fe37aa9741a9b892cf3103b1b6de deleted and replaced by tablet 0963f28287b3483daca2b517f2b2c7f5
I20251211 08:58:38.520864 17558 consensus_queue.cc:260] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:38.522825 17558 ts_tablet_manager.cc:1434] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251211 08:58:38.523787 17553 raft_consensus.cc:740] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.524608 17553 consensus_queue.cc:260] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.527196 17558 tablet_bootstrap.cc:492] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
I20251211 08:58:38.533481 17555 tablet_bootstrap.cc:492] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
I20251211 08:58:38.538200 17558 tablet_bootstrap.cc:654] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.538727 17555 tablet_bootstrap.cc:654] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.531215 17276 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bb2fe37aa9741a9b892cf3103b1b6de with delete_type TABLET_DATA_DELETED (replaced by tablet 0963f28287b3483daca2b517f2b2c7f5) from {username='slave'} at 127.0.0.1:37278
I20251211 08:58:38.541394 17565 ts_tablet_manager.cc:1916] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:38.556732 17558 tablet_bootstrap.cc:492] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:38.557269 17558 ts_tablet_manager.cc:1403] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.001s
I20251211 08:58:38.558936 17564 ts_tablet_manager.cc:1929] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:38.559578 17558 raft_consensus.cc:359] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.559875 17117 tablet_service.cc:1505] Processing CreateTablet for tablet 0963f28287b3483daca2b517f2b2c7f5 (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.560286 17558 raft_consensus.cc:385] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.560565 17558 raft_consensus.cc:740] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.560822 17276 tablet_service.cc:1505] Processing CreateTablet for tablet 0963f28287b3483daca2b517f2b2c7f5 (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.562036 17429 tablet_service.cc:1505] Processing CreateTablet for tablet 0963f28287b3483daca2b517f2b2c7f5 (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.563495 17429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0963f28287b3483daca2b517f2b2c7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.563668 17276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0963f28287b3483daca2b517f2b2c7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.564319 17553 ts_tablet_manager.cc:1434] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.092s user 0.005s sys 0.000s
I20251211 08:58:38.565912 17558 consensus_queue.cc:260] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.571056 17117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0963f28287b3483daca2b517f2b2c7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.572106 17558 ts_tablet_manager.cc:1434] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.015s user 0.007s sys 0.001s
I20251211 08:58:38.574271 17558 tablet_bootstrap.cc:492] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
08:58:38.574 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_1bf93acd2af64f98b4cda7ce375e702e already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20251211 08:58:38.582297 16848 catalog_manager.cc:2170] Already present: failed to create HMS catalog entry for table [id=48ee8379825f4a37817af266a3e9d216]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_1bf93acd2af64f98b4cda7ce375e702e already exists)
I20251211 08:58:38.587446 17568 raft_consensus.cc:493] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:38.588003 17568 raft_consensus.cc:515] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.588987 17558 tablet_bootstrap.cc:654] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.593281 17568 leader_election.cc:290] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
W20251211 08:58:38.599620 17190 tablet.cc:2378] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251211 08:58:38.600091 17553 tablet_bootstrap.cc:492] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:38.606489 16848 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39386:
name: "default.Table_7bc946992a8e44a0987ff88263b4c255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:38.618045 17553 tablet_bootstrap.cc:654] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.618172 17564 log.cc:1199] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/0a6e1fd3f0e04978b128695735e5d038
I20251211 08:58:38.619012 17564 ts_tablet_manager.cc:1950] T 0a6e1fd3f0e04978b128695735e5d038 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting consensus metadata
I20251211 08:58:38.621496 17564 tablet_replica.cc:333] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf: stopping tablet replica
I20251211 08:58:38.621556 16836 catalog_manager.cc:4985] TS 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): tablet 0a6e1fd3f0e04978b128695735e5d038 (table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]) successfully deleted
I20251211 08:58:38.622177 17564 raft_consensus.cc:2243] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:38.624615 17558 tablet_bootstrap.cc:492] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:38.625854 17558 ts_tablet_manager.cc:1403] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.007s
I20251211 08:58:38.627446 17563 ts_tablet_manager.cc:1929] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:38.627851 17563 log.cc:1199] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/0a6e1fd3f0e04978b128695735e5d038
I20251211 08:58:38.628661 17563 ts_tablet_manager.cc:1950] T 0a6e1fd3f0e04978b128695735e5d038 P 62329f65aae54e84b396a51dd81f1b70: Deleting consensus metadata
I20251211 08:58:38.640103 17449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0bb2fe37aa9741a9b892cf3103b1b6de" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" is_pre_election: true
I20251211 08:58:38.640870 17558 raft_consensus.cc:359] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.641531 17558 raft_consensus.cc:385] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.641803 17558 raft_consensus.cc:740] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.642477 17558 consensus_queue.cc:260] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.644675 17558 ts_tablet_manager.cc:1434] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.018s user 0.002s sys 0.003s
I20251211 08:58:38.645550 17558 tablet_bootstrap.cc:492] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
I20251211 08:58:38.648700 16835 catalog_manager.cc:4985] TS 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): tablet 0a6e1fd3f0e04978b128695735e5d038 (table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]) successfully deleted
I20251211 08:58:38.649574 17563 tablet_replica.cc:333] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70: stopping tablet replica
I20251211 08:58:38.650346 17563 raft_consensus.cc:2243] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:38.662832 17563 raft_consensus.cc:2272] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:38.664886 17563 ts_tablet_manager.cc:1916] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:38.670372 17558 tablet_bootstrap.cc:654] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.660552 17565 ts_tablet_manager.cc:1929] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:38.671634 17565 log.cc:1199] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/e6e62fc302554a84bcb15f0c8f97cba5
I20251211 08:58:38.672309 17565 ts_tablet_manager.cc:1950] T e6e62fc302554a84bcb15f0c8f97cba5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting consensus metadata
I20251211 08:58:38.622673 17564 raft_consensus.cc:2272] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:38.677023 17564 ts_tablet_manager.cc:1916] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:38.677580 17553 tablet_bootstrap.cc:492] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:38.677945 17553 ts_tablet_manager.cc:1403] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.078s user 0.008s sys 0.001s
I20251211 08:58:38.680464 17553 raft_consensus.cc:359] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.681121 17553 raft_consensus.cc:385] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.681365 17553 raft_consensus.cc:740] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.683012 17553 consensus_queue.cc:260] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
W20251211 08:58:38.686131 17071 leader_election.cc:343] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:38.689291 16835 catalog_manager.cc:4985] TS e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): tablet e6e62fc302554a84bcb15f0c8f97cba5 (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]) successfully deleted
I20251211 08:58:38.690181 17565 tablet_replica.cc:333] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2: stopping tablet replica
I20251211 08:58:38.692816 16845 master_service.cc:954] ReplaceTablet: received request to replace tablet 0bb2fe37aa9741a9b892cf3103b1b6de from {username='slave'} at 127.0.0.1:39386
I20251211 08:58:38.695493 17555 tablet_bootstrap.cc:492] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
I20251211 08:58:38.695899 17555 ts_tablet_manager.cc:1403] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.163s user 0.009s sys 0.000s
I20251211 08:58:38.696592 17558 tablet_bootstrap.cc:492] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:38.697017 17558 ts_tablet_manager.cc:1403] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.001s
I20251211 08:58:38.699668 17558 raft_consensus.cc:359] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.700361 17558 raft_consensus.cc:385] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.700649 17558 raft_consensus.cc:740] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.700541 17296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0bb2fe37aa9741a9b892cf3103b1b6de" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" is_pre_election: true
I20251211 08:58:38.704051 17558 consensus_queue.cc:260] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.698417 17555 raft_consensus.cc:359] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.708266 17555 raft_consensus.cc:385] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.708532 17555 raft_consensus.cc:740] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.709296 17555 consensus_queue.cc:260] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.709929 17553 ts_tablet_manager.cc:1434] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20251211 08:58:38.690734 17565 raft_consensus.cc:2243] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:38.710736 17565 raft_consensus.cc:2272] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:38.712785 17558 ts_tablet_manager.cc:1434] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
W20251211 08:58:38.723260 17070 leader_election.cc:343] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:38.723668 17070 leader_election.cc:304] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70; no voters: 7467cf4890bc486bbfb9b8f4bea2c2cf, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 08:58:38.727033 17568 raft_consensus.cc:2749] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:38.729635 17565 ts_tablet_manager.cc:1916] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:38.730258 16845 master_service.cc:954] ReplaceTablet: received request to replace tablet 0bb2fe37aa9741a9b892cf3103b1b6de from {username='slave'} at 127.0.0.1:39386
I20251211 08:58:38.733280 17555 ts_tablet_manager.cc:1434] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.037s user 0.003s sys 0.004s
I20251211 08:58:38.733385 17553 tablet_bootstrap.cc:492] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:38.747196 17555 tablet_bootstrap.cc:492] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
I20251211 08:58:38.751209 17563 ts_tablet_manager.cc:1929] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:38.751585 17563 log.cc:1199] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/e6e62fc302554a84bcb15f0c8f97cba5
I20251211 08:58:38.752221 17563 ts_tablet_manager.cc:1950] T e6e62fc302554a84bcb15f0c8f97cba5 P 62329f65aae54e84b396a51dd81f1b70: Deleting consensus metadata
I20251211 08:58:38.753258 17555 tablet_bootstrap.cc:654] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.754453 16835 catalog_manager.cc:4985] TS 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): tablet e6e62fc302554a84bcb15f0c8f97cba5 (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]) successfully deleted
I20251211 08:58:38.756747 17553 tablet_bootstrap.cc:654] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.759318 17562 raft_consensus.cc:493] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:38.759821 17562 raft_consensus.cc:515] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.761941 17564 ts_tablet_manager.cc:1929] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:38.762352 17564 log.cc:1199] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/0bb2fe37aa9741a9b892cf3103b1b6de
I20251211 08:58:38.767191 17562 leader_election.cc:290] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:38.774156 16851 catalog_manager.cc:2174] created HMS catalog entry for table [id=9d454284fb154c6297fc7ea77ed02283]
I20251211 08:58:38.779311 17563 tablet_replica.cc:333] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70: stopping tablet replica
I20251211 08:58:38.780220 17563 raft_consensus.cc:2243] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:38.780696 17563 raft_consensus.cc:2272] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:38.783447 17563 ts_tablet_manager.cc:1916] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:38.791507 16851 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39386 (request call id 17) took 1098 ms. Trace:
I20251211 08:58:38.795885 17564 ts_tablet_manager.cc:1950] T 0bb2fe37aa9741a9b892cf3103b1b6de P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting consensus metadata
I20251211 08:58:38.798221 16836 catalog_manager.cc:4985] TS 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): tablet 0bb2fe37aa9741a9b892cf3103b1b6de (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]) successfully deleted
I20251211 08:58:38.791867 16851 rpcz_store.cc:276] 1211 08:58:37.693307 (+ 0us) service_pool.cc:168] Inserting onto call queue
1211 08:58:37.693481 (+ 174us) service_pool.cc:225] Handling call
1211 08:58:37.709663 (+ 16182us) catalog_manager.cc:2101] Acquired catalog manager lock
1211 08:58:37.723126 (+ 13463us) catalog_manager.cc:2147] Created new table and tablet info
1211 08:58:38.774115 (+1050989us) catalog_manager.cc:2173] Created new table in HMS catalog
1211 08:58:38.790726 (+ 16611us) catalog_manager.cc:2207] Wrote table and tablets to system table
1211 08:58:38.791124 (+ 398us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1211 08:58:38.791442 (+ 318us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1211 08:58:38.776142 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1211 08:58:38.776704 (+ 562us) write_op.cc:432] Acquiring schema lock in shared mode
1211 08:58:38.776738 (+ 34us) write_op.cc:435] Acquired schema lock
1211 08:58:38.776762 (+ 24us) tablet.cc:660] Decoding operations
1211 08:58:38.777206 (+ 444us) write_op.cc:620] Acquiring the partition lock for write op
1211 08:58:38.777281 (+ 75us) write_op.cc:641] Partition lock acquired for write op
1211 08:58:38.777318 (+ 37us) tablet.cc:683] Acquiring locks for 2 operations
1211 08:58:38.777598 (+ 280us) tablet.cc:699] Row locks acquired
1211 08:58:38.777622 (+ 24us) write_op.cc:260] PREPARE: finished
1211 08:58:38.777750 (+ 128us) write_op.cc:270] Start()
1211 08:58:38.777836 (+ 86us) write_op.cc:276] Timestamp: P: 1765443518777821 usec, L: 0
1211 08:58:38.777853 (+ 17us) op_driver.cc:348] REPLICATION: starting
1211 08:58:38.778392 (+ 539us) log.cc:844] Serialized 432 byte log entry
1211 08:58:38.787861 (+ 9469us) op_driver.cc:464] REPLICATION: finished
1211 08:58:38.788200 (+ 339us) write_op.cc:301] APPLY: starting
1211 08:58:38.788271 (+ 71us) tablet.cc:1365] starting BulkCheckPresence
1211 08:58:38.788382 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
1211 08:58:38.788464 (+ 82us) tablet.cc:1370] starting ApplyRowOperation cycle
1211 08:58:38.788825 (+ 361us) tablet.cc:1381] finished ApplyRowOperation cycle
1211 08:58:38.788965 (+ 140us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1211 08:58:38.789021 (+ 56us) write_op.cc:312] APPLY: finished
1211 08:58:38.789219 (+ 198us) log.cc:844] Serialized 28 byte log entry
1211 08:58:38.789357 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:38.789425 (+ 68us) write_op.cc:454] Released schema lock
1211 08:58:38.789463 (+ 38us) write_op.cc:341] FINISH: Updating metrics
1211 08:58:38.789695 (+ 232us) write_op.cc:489] Releasing partition, row and schema locks
1211 08:58:38.789723 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":857981,"Hive Metastore.run_cpu_time_us":567,"Hive Metastore.run_wall_time_us":192180,"child_traces":[["op",{"apply.queue_time_us":181,"apply.run_cpu_time_us":1451,"apply.run_wall_time_us":1446,"num_ops":2,"prepare.queue_time_us":138,"prepare.run_cpu_time_us":2619,"prepare.run_wall_time_us":2619,"raft.queue_time_us":2629,"raft.run_cpu_time_us":658,"raft.run_wall_time_us":655,"replication_time_us":9897}]]}
I20251211 08:58:38.802074 17553 tablet_bootstrap.cc:492] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:38.802479 17553 ts_tablet_manager.cc:1403] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.005s
I20251211 08:58:38.804400 16851 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39386:
table { table_name: "default.Table_12ea97bf52084b09955460036752d6e4" } modify_external_catalogs: true
I20251211 08:58:38.804869 17553 raft_consensus.cc:359] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.805120 16851 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39386:
table { table_name: "default.Table_12ea97bf52084b09955460036752d6e4" } modify_external_catalogs: true
I20251211 08:58:38.805456 17553 raft_consensus.cc:385] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.808003 17553 raft_consensus.cc:740] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.813232 17138 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a697ef839f1045a39f8621e5cd9f7233" candidate_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62329f65aae54e84b396a51dd81f1b70" is_pre_election: true
I20251211 08:58:38.814015 17138 raft_consensus.cc:2468] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8a2e7dba69c4c80bb3672eeaf63e1c2 in term 0.
I20251211 08:58:38.814458 17555 tablet_bootstrap.cc:492] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
I20251211 08:58:38.815153 17555 ts_tablet_manager.cc:1403] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.068s user 0.011s sys 0.000s
I20251211 08:58:38.815220 17229 leader_election.cc:304] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2; no voters:
I20251211 08:58:38.816069 17562 raft_consensus.cc:2804] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:38.816397 17562 raft_consensus.cc:493] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:38.816697 17562 raft_consensus.cc:3060] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:38.817644 17555 raft_consensus.cc:359] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.818359 17555 raft_consensus.cc:385] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.818666 17555 raft_consensus.cc:740] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.820981 17565 ts_tablet_manager.cc:1929] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:38.821332 17565 log.cc:1199] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/0bb2fe37aa9741a9b892cf3103b1b6de
I20251211 08:58:38.821995 17565 ts_tablet_manager.cc:1950] T 0bb2fe37aa9741a9b892cf3103b1b6de P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting consensus metadata
I20251211 08:58:38.823348 17562 raft_consensus.cc:515] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.824520 16835 catalog_manager.cc:4985] TS e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): tablet 0bb2fe37aa9741a9b892cf3103b1b6de (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]) successfully deleted
I20251211 08:58:38.826113 17553 consensus_queue.cc:260] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.826448 17555 consensus_queue.cc:260] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:38.827178 17138 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a697ef839f1045a39f8621e5cd9f7233" candidate_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62329f65aae54e84b396a51dd81f1b70"
I20251211 08:58:38.827728 17138 raft_consensus.cc:3060] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:38.834568 17449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a697ef839f1045a39f8621e5cd9f7233" candidate_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" is_pre_election: true
I20251211 08:58:38.835256 17449 raft_consensus.cc:2468] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8a2e7dba69c4c80bb3672eeaf63e1c2 in term 0.
I20251211 08:58:38.837165 17138 raft_consensus.cc:2468] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8a2e7dba69c4c80bb3672eeaf63e1c2 in term 1.
I20251211 08:58:38.838442 17229 leader_election.cc:304] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2; no voters:
F20251211 08:58:38.841133 17009 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1765443517, eventType=CREATE_TABLE, dbName=default, tableName=table_1bf93acd2af64f98b4cda7ce375e702e, message={"server":"","servicePrincipal":"","db":"default","table":"table_1bf93acd2af64f98b4cda7ce375e702e","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_1bf93acd2af64f98b4cda7ce375e702e\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765443517},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/warehouse/table_1bf93acd2af64f98b4cda7ce375e702e\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_1bf93acd2af64f98b4cda7ce375e702e\",\"kudu.table_id\":\"86b322f346e0458daf6c4f9dd962de20\",\"transient_lastDdlTime\":\"1765443517\",\"comment\":\"\",\"kudu.master_addresses\":\"127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"c258c1a306e24c4197d31ee43870f1fb\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765443517,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765443517, eventType=CREATE_TABLE, dbName=default, tableName=table_d36118f54fec423badd3d2b59d64b05c, message={"server":"","servicePrincipal":"","db":"default","table":"table_d36118f54fec423badd3d2b59d64b05c","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d36118f54fec423badd3d2b59d64b05c\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765443517},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/warehouse/table_d36118f54fec423badd3d2b59d64b05c\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d36118f54fec423badd3d2b59d64b05c\",\"kudu.table_id\":\"53d392cd36244b498c2740a372a1daf4\",\"transient_lastDdlTime\":\"1765443517\",\"comment\":\"\",\"kudu.master_addresses\":\"127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"c258c1a306e24c4197d31ee43870f1fb\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765443517,"files":[]}, messageFormat=json-0.2, catName=hive);
I20251211 08:58:38.843240 17555 ts_tablet_manager.cc:1434] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20251211 08:58:38.843922 17563 ts_tablet_manager.cc:1929] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:38.844352 17563 log.cc:1199] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/0bb2fe37aa9741a9b892cf3103b1b6de
I20251211 08:58:38.845045 17563 ts_tablet_manager.cc:1950] T 0bb2fe37aa9741a9b892cf3103b1b6de P 62329f65aae54e84b396a51dd81f1b70: Deleting consensus metadata
I20251211 08:58:38.855181 17562 leader_election.cc:290] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 election: Requested vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:38.855487 17449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a697ef839f1045a39f8621e5cd9f7233" candidate_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf"
I20251211 08:58:38.856027 17449 raft_consensus.cc:3060] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:38.859905 17562 raft_consensus.cc:2804] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:38.860483 17562 raft_consensus.cc:697] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 LEADER]: Becoming Leader. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Running, Role: LEADER
I20251211 08:58:38.861330 17562 consensus_queue.cc:237] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
*** Check failure stack trace: ***
I20251211 08:58:38.848469 16835 catalog_manager.cc:4985] TS 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): tablet 0bb2fe37aa9741a9b892cf3103b1b6de (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]) successfully deleted
I20251211 08:58:38.878360 17449 raft_consensus.cc:2468] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8a2e7dba69c4c80bb3672eeaf63e1c2 in term 1.
I20251211 08:58:38.890148 17276 tablet_service.cc:1505] Processing CreateTablet for tablet 61ae46a42d1e46e3a4754213c2d19259 (DEFAULT_TABLE table=default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.891426 17276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ae46a42d1e46e3a4754213c2d19259. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.904127 17117 tablet_service.cc:1505] Processing CreateTablet for tablet 61ae46a42d1e46e3a4754213c2d19259 (DEFAULT_TABLE table=default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.905272 17117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ae46a42d1e46e3a4754213c2d19259. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.908313 16844 master_service.cc:954] ReplaceTablet: received request to replace tablet a697ef839f1045a39f8621e5cd9f7233 from {username='slave'} at 127.0.0.1:39386
I20251211 08:58:38.919939 17558 tablet_bootstrap.cc:492] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
W20251211 08:58:38.924849 17229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.146.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 86563.8us
I20251211 08:58:38.937659 17558 tablet_bootstrap.cc:654] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.940269 17553 ts_tablet_manager.cc:1434] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.138s user 0.013s sys 0.068s
I20251211 08:58:38.942191 17553 tablet_bootstrap.cc:492] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:38.942111 16843 master_service.cc:954] ReplaceTablet: received request to replace tablet 0963f28287b3483daca2b517f2b2c7f5 from {username='slave'} at 127.0.0.1:39386
I20251211 08:58:38.951771 16844 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Sending DeleteTablet for 3 replicas of tablet a697ef839f1045a39f8621e5cd9f7233
I20251211 08:58:38.952632 17553 tablet_bootstrap.cc:654] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:38.953498 17118 tablet_service.cc:1552] Processing DeleteTablet for tablet a697ef839f1045a39f8621e5cd9f7233 with delete_type TABLET_DATA_DELETED (replaced by tablet 81b5f03965634d6db4b2c1737a26bf30) from {username='slave'} at 127.0.0.1:43290
I20251211 08:58:38.882861 17429 tablet_service.cc:1505] Processing CreateTablet for tablet 61ae46a42d1e46e3a4754213c2d19259 (DEFAULT_TABLE table=default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:38.954341 17563 tablet_replica.cc:333] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70: stopping tablet replica
I20251211 08:58:38.954998 17563 raft_consensus.cc:2243] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:38.955145 17429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ae46a42d1e46e3a4754213c2d19259. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:38.955400 17563 raft_consensus.cc:2272] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:38.965227 17563 ts_tablet_manager.cc:1916] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:38.967536 17558 tablet_bootstrap.cc:492] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:38.968086 17558 ts_tablet_manager.cc:1403] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.011s
I20251211 08:58:38.971220 17558 raft_consensus.cc:359] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } }
I20251211 08:58:38.971944 17558 raft_consensus.cc:385] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:38.972201 17558 raft_consensus.cc:740] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:38.972956 17558 consensus_queue.cc:260] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } }
I20251211 08:58:38.980628 17558 ts_tablet_manager.cc:1434] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.012s user 0.007s sys 0.001s
I20251211 08:58:38.997638 17568 raft_consensus.cc:493] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:38.998252 17568 raft_consensus.cc:515] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } }
I20251211 08:58:39.000564 17568 leader_election.cc:290] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:39.002223 17296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ae46a42d1e46e3a4754213c2d19259" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" is_pre_election: true
I20251211 08:58:39.003072 17555 tablet_bootstrap.cc:492] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
W20251211 08:58:39.006755 17070 leader_election.cc:343] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Illegal state: must be running to vote when last-logged opid is not known
I20251211 08:58:39.008750 17555 tablet_bootstrap.cc:654] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:39.015564 17553 tablet_bootstrap.cc:492] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:39.016021 17553 ts_tablet_manager.cc:1403] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.074s user 0.038s sys 0.014s
I20251211 08:58:39.017669 17555 tablet_bootstrap.cc:492] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
I20251211 08:58:39.018097 17555 ts_tablet_manager.cc:1403] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20251211 08:58:39.018509 17553 raft_consensus.cc:359] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } }
I20251211 08:58:39.019241 17553 raft_consensus.cc:385] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:39.019519 17553 raft_consensus.cc:740] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
I20251211 08:58:39.020179 17553 consensus_queue.cc:260] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } }
I20251211 08:58:39.020651 17555 raft_consensus.cc:359] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } }
I20251211 08:58:39.021778 17555 raft_consensus.cc:385] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:39.022071 17555 raft_consensus.cc:740] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:39.022758 17553 ts_tablet_manager.cc:1434] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251211 08:58:39.022841 17555 consensus_queue.cc:260] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } }
I20251211 08:58:39.021776 16842 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39386:
name: "default.Table_9088340c00c94ec8bb9312a333dec04d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:39.028676 17449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ae46a42d1e46e3a4754213c2d19259" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" is_pre_election: true
I20251211 08:58:39.029376 16843 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Sending DeleteTablet for 3 replicas of tablet 0963f28287b3483daca2b517f2b2c7f5
I20251211 08:58:39.030040 17449 raft_consensus.cc:2468] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62329f65aae54e84b396a51dd81f1b70 in term 0.
I20251211 08:58:39.030453 16844 catalog_manager.cc:6724] ReplaceTablet: tablet a697ef839f1045a39f8621e5cd9f7233 deleted and replaced by tablet 81b5f03965634d6db4b2c1737a26bf30
I20251211 08:58:39.031530 17276 tablet_service.cc:1552] Processing DeleteTablet for tablet a697ef839f1045a39f8621e5cd9f7233 with delete_type TABLET_DATA_DELETED (replaced by tablet 81b5f03965634d6db4b2c1737a26bf30) from {username='slave'} at 127.0.0.1:37278
I20251211 08:58:39.032294 17565 tablet_replica.cc:333] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: stopping tablet replica
I20251211 08:58:39.032289 17428 tablet_service.cc:1552] Processing DeleteTablet for tablet 0963f28287b3483daca2b517f2b2c7f5 with delete_type TABLET_DATA_DELETED (replaced by tablet cb7a90c0accf4669a1cdc4591284954f) from {username='slave'} at 127.0.0.1:52092
I20251211 08:58:39.030351 17429 tablet_service.cc:1552] Processing DeleteTablet for tablet a697ef839f1045a39f8621e5cd9f7233 with delete_type TABLET_DATA_DELETED (replaced by tablet 81b5f03965634d6db4b2c1737a26bf30) from {username='slave'} at 127.0.0.1:52092
I20251211 08:58:39.033032 17565 raft_consensus.cc:2243] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:39.033568 17071 leader_election.cc:304] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 08:58:39.033809 17565 pending_rounds.cc:70] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Trying to abort 1 pending ops.
I20251211 08:58:39.034044 17565 pending_rounds.cc:77] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231256653529497600 op_type: NO_OP noop_request { }
I20251211 08:58:39.034477 17565 raft_consensus.cc:2889] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251211 08:58:39.034644 17568 raft_consensus.cc:2804] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:39.034849 17565 raft_consensus.cc:2272] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:39.035032 17568 raft_consensus.cc:493] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:39.035338 17568 raft_consensus.cc:3060] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:39.035501 17564 tablet_replica.cc:333] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: stopping tablet replica
I20251211 08:58:39.038012 17565 ts_tablet_manager.cc:1916] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fa498111950 google::LogMessage::Flush() at ??:0
I20251211 08:58:39.030757 16843 catalog_manager.cc:6724] ReplaceTablet: tablet 0963f28287b3483daca2b517f2b2c7f5 deleted and replaced by tablet cb7a90c0accf4669a1cdc4591284954f
I20251211 08:58:39.041785 17568 raft_consensus.cc:515] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } }
I20251211 08:58:39.049196 17563 ts_tablet_manager.cc:1929] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:39.049542 17563 log.cc:1199] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/a697ef839f1045a39f8621e5cd9f7233
I20251211 08:58:39.050539 17563 ts_tablet_manager.cc:1950] T a697ef839f1045a39f8621e5cd9f7233 P 62329f65aae54e84b396a51dd81f1b70: Deleting consensus metadata
I20251211 08:58:39.053416 17117 tablet_service.cc:1552] Processing DeleteTablet for tablet 0963f28287b3483daca2b517f2b2c7f5 with delete_type TABLET_DATA_DELETED (replaced by tablet cb7a90c0accf4669a1cdc4591284954f) from {username='slave'} at 127.0.0.1:43290
I20251211 08:58:39.054994 17276 tablet_service.cc:1552] Processing DeleteTablet for tablet 0963f28287b3483daca2b517f2b2c7f5 with delete_type TABLET_DATA_DELETED (replaced by tablet cb7a90c0accf4669a1cdc4591284954f) from {username='slave'} at 127.0.0.1:37278
I20251211 08:58:39.055042 16835 catalog_manager.cc:4985] TS 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): tablet a697ef839f1045a39f8621e5cd9f7233 (table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]) successfully deleted
I20251211 08:58:39.056625 17296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ae46a42d1e46e3a4754213c2d19259" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2"
I20251211 08:58:39.056916 17449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ae46a42d1e46e3a4754213c2d19259" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf"
I20251211 08:58:39.057183 17296 raft_consensus.cc:3060] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:39.057461 17449 raft_consensus.cc:3060] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Advancing to term 1
@ 0x7fa4981116bf google::LogMessage::~LogMessage() at ??:0
I20251211 08:58:39.064198 17449 raft_consensus.cc:2468] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62329f65aae54e84b396a51dd81f1b70 in term 1.
I20251211 08:58:39.064421 17296 raft_consensus.cc:2468] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62329f65aae54e84b396a51dd81f1b70 in term 1.
I20251211 08:58:39.065243 17071 leader_election.cc:304] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters:
I20251211 08:58:39.065879 17563 tablet_replica.cc:333] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70: stopping tablet replica
I20251211 08:58:39.066568 17563 raft_consensus.cc:2243] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:39.067005 17563 raft_consensus.cc:2272] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:39.067463 17565 ts_tablet_manager.cc:1929] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:39.067806 17565 log.cc:1199] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/a697ef839f1045a39f8621e5cd9f7233
I20251211 08:58:39.068503 17565 ts_tablet_manager.cc:1950] T a697ef839f1045a39f8621e5cd9f7233 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting consensus metadata
I20251211 08:58:39.069900 17564 raft_consensus.cc:2243] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:39.070356 17564 raft_consensus.cc:2272] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:39.072341 17555 ts_tablet_manager.cc:1434] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20251211 08:58:39.072381 17564 ts_tablet_manager.cc:1916] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:39.074877 17568 leader_election.cc:290] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 election: Requested vote from peers e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:39.078940 17592 raft_consensus.cc:2804] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:39.079505 17592 raft_consensus.cc:697] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 1 LEADER]: Becoming Leader. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Running, Role: LEADER
@ 0x7fa4a31a426c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251211 08:58:39.080412 17592 consensus_queue.cc:237] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } }
I20251211 08:58:39.089834 17429 tablet_service.cc:1505] Processing CreateTablet for tablet 81b5f03965634d6db4b2c1737a26bf30 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:39.090946 17429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b5f03965634d6db4b2c1737a26bf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:39.091917 17565 tablet_replica.cc:333] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: stopping tablet replica
I20251211 08:58:39.092656 17565 raft_consensus.cc:2243] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:39.093116 17565 raft_consensus.cc:2272] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:39.093647 17117 tablet_service.cc:1505] Processing CreateTablet for tablet 81b5f03965634d6db4b2c1737a26bf30 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:39.094704 17117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b5f03965634d6db4b2c1737a26bf30. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fa4a31a3a2c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251211 08:58:39.105839 17276 tablet_service.cc:1505] Processing CreateTablet for tablet 81b5f03965634d6db4b2c1737a26bf30 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:39.106902 17564 ts_tablet_manager.cc:1929] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:39.107153 17276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b5f03965634d6db4b2c1737a26bf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:39.107288 17564 log.cc:1199] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/0963f28287b3483daca2b517f2b2c7f5
I20251211 08:58:39.107267 16835 catalog_manager.cc:4985] TS e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): tablet a697ef839f1045a39f8621e5cd9f7233 (table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]) successfully deleted
I20251211 08:58:39.107914 17564 ts_tablet_manager.cc:1950] T 0963f28287b3483daca2b517f2b2c7f5 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting consensus metadata
I20251211 08:58:39.110505 17564 tablet_replica.cc:333] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf: stopping tablet replica
I20251211 08:58:39.110486 16836 catalog_manager.cc:4985] TS 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): tablet 0963f28287b3483daca2b517f2b2c7f5 (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]) successfully deleted
I20251211 08:58:39.116168 17565 ts_tablet_manager.cc:1916] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:39.111284 17564 raft_consensus.cc:2243] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:39.120543 17564 raft_consensus.cc:2272] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fa4a31a7aa2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251211 08:58:39.126848 17563 ts_tablet_manager.cc:1916] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:39.135157 17558 tablet_bootstrap.cc:492] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
I20251211 08:58:39.122699 17564 ts_tablet_manager.cc:1916] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:39.140744 17558 tablet_bootstrap.cc:654] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:39.150506 17558 tablet_bootstrap.cc:492] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:39.150959 17558 ts_tablet_manager.cc:1403] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20251211 08:58:39.153589 17555 tablet_bootstrap.cc:492] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
I20251211 08:58:39.153285 17558 raft_consensus.cc:359] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.153842 17558 raft_consensus.cc:385] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:39.154064 17558 raft_consensus.cc:740] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:39.154706 17558 consensus_queue.cc:260] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.164350 17275 tablet_service.cc:1552] Processing DeleteTablet for tablet a697ef839f1045a39f8621e5cd9f7233 with delete_type TABLET_DATA_DELETED (replaced by tablet 81b5f03965634d6db4b2c1737a26bf30) from {username='slave'} at 127.0.0.1:37278
I20251211 08:58:39.164912 17555 tablet_bootstrap.cc:654] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:39.165498 17558 ts_tablet_manager.cc:1434] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
@ 0x7fa4a31a7a5a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20251211 08:58:39.180688 17553 tablet_bootstrap.cc:492] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:39.184024 17564 ts_tablet_manager.cc:1929] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:39.184402 17564 log.cc:1199] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/a697ef839f1045a39f8621e5cd9f7233
I20251211 08:58:39.185057 17564 ts_tablet_manager.cc:1950] T a697ef839f1045a39f8621e5cd9f7233 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting consensus metadata
@ 0x7fa4a31a79ea std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20251211 08:58:39.187311 16836 catalog_manager.cc:4985] TS 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): tablet a697ef839f1045a39f8621e5cd9f7233 (table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]) successfully deleted
I20251211 08:58:39.189018 17553 tablet_bootstrap.cc:654] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:39.192359 16813 master_runner.cc:186] Error getting master registration for 127.13.146.60:34701: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111),
I20251211 08:58:39.192837 16813 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:58:39.194911 17555 tablet_bootstrap.cc:492] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
I20251211 08:58:39.200191 17553 tablet_bootstrap.cc:492] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:39.200584 17553 ts_tablet_manager.cc:1403] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20251211 08:58:39.203570 17555 ts_tablet_manager.cc:1403] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.003s
I20251211 08:58:39.203289 17553 raft_consensus.cc:359] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.206233 17553 raft_consensus.cc:385] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:39.206499 17553 raft_consensus.cc:740] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
@ 0x7fa4a31a79b2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20251211 08:58:39.213078 17555 raft_consensus.cc:359] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.213732 17555 raft_consensus.cc:385] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:39.213981 17555 raft_consensus.cc:740] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:39.214536 17555 consensus_queue.cc:260] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.216414 17555 ts_tablet_manager.cc:1434] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20251211 08:58:39.219833 17563 ts_tablet_manager.cc:1929] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:39.220208 17563 log.cc:1199] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/0963f28287b3483daca2b517f2b2c7f5
I20251211 08:58:39.220938 17563 ts_tablet_manager.cc:1950] T 0963f28287b3483daca2b517f2b2c7f5 P 62329f65aae54e84b396a51dd81f1b70: Deleting consensus metadata
I20251211 08:58:39.214697 17553 consensus_queue.cc:260] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.223250 17553 ts_tablet_manager.cc:1434] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20251211 08:58:39.227937 17117 tablet_service.cc:1505] Processing CreateTablet for tablet cb7a90c0accf4669a1cdc4591284954f (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:39.229046 17117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb7a90c0accf4669a1cdc4591284954f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:39.231913 17565 ts_tablet_manager.cc:1929] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:39.232230 17565 log.cc:1199] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/0963f28287b3483daca2b517f2b2c7f5
I20251211 08:58:39.232833 17565 ts_tablet_manager.cc:1950] T 0963f28287b3483daca2b517f2b2c7f5 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting consensus metadata
I20251211 08:58:39.235616 16835 catalog_manager.cc:4985] TS 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): tablet 0963f28287b3483daca2b517f2b2c7f5 (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]) successfully deleted
I20251211 08:58:39.237476 17276 tablet_service.cc:1505] Processing CreateTablet for tablet cb7a90c0accf4669a1cdc4591284954f (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:39.240058 17429 tablet_service.cc:1505] Processing CreateTablet for tablet cb7a90c0accf4669a1cdc4591284954f (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:39.241094 17429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb7a90c0accf4669a1cdc4591284954f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:39.245540 16844 catalog_manager.cc:5654] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1). New cstate: current_term: 1 leader_uuid: "62329f65aae54e84b396a51dd81f1b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } health_report { overall_health: HEALTHY } } }
I20251211 08:58:39.248672 17276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb7a90c0accf4669a1cdc4591284954f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:39.255224 16845 master_service.cc:954] ReplaceTablet: received request to replace tablet 81b5f03965634d6db4b2c1737a26bf30 from {username='slave'} at 127.0.0.1:39386
I20251211 08:58:39.264214 16835 catalog_manager.cc:4985] TS e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): tablet 0963f28287b3483daca2b517f2b2c7f5 (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]) successfully deleted
W20251211 08:58:39.264923 16835 catalog_manager.cc:4952] TS e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): delete failed for tablet a697ef839f1045a39f8621e5cd9f7233 because the tablet was not found. No further retry: Not found: Tablet not found: a697ef839f1045a39f8621e5cd9f7233
@ 0x7fa4a31a6cae std::__1::__function::__func<>::operator()() at ??:0
I20251211 08:58:39.271453 17555 tablet_bootstrap.cc:492] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
@ 0x5aa2b5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5aa0ea std::__1::function<>::operator()() at ??:?
I20251211 08:58:39.275184 17553 tablet_bootstrap.cc:492] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:39.276401 17555 tablet_bootstrap.cc:654] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:39.280076 17553 tablet_bootstrap.cc:654] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:39.296700 16845 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Sending DeleteTablet for 3 replicas of tablet 81b5f03965634d6db4b2c1737a26bf30
I20251211 08:58:39.298281 16845 catalog_manager.cc:6724] ReplaceTablet: tablet 81b5f03965634d6db4b2c1737a26bf30 deleted and replaced by tablet 316fbbd2f6e0476e9b9768b954a21fbc
I20251211 08:58:39.300310 17553 tablet_bootstrap.cc:492] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:39.300702 17553 ts_tablet_manager.cc:1403] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.010s
I20251211 08:58:39.301139 17429 tablet_service.cc:1552] Processing DeleteTablet for tablet 81b5f03965634d6db4b2c1737a26bf30 with delete_type TABLET_DATA_DELETED (replaced by tablet 316fbbd2f6e0476e9b9768b954a21fbc) from {username='slave'} at 127.0.0.1:52092
I20251211 08:58:39.301973 17564 tablet_replica.cc:333] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf: stopping tablet replica
I20251211 08:58:39.302692 17564 raft_consensus.cc:2243] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:39.303244 17564 raft_consensus.cc:2272] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:39.303059 17553 raft_consensus.cc:359] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.303709 17553 raft_consensus.cc:385] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:39.303939 17553 raft_consensus.cc:740] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
I20251211 08:58:39.305382 17553 consensus_queue.cc:260] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.307119 17553 ts_tablet_manager.cc:1434] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251211 08:58:39.314692 16843 master_service.cc:954] ReplaceTablet: received request to replace tablet 81b5f03965634d6db4b2c1737a26bf30 from {username='slave'} at 127.0.0.1:39386
I20251211 08:58:39.316277 17558 tablet_bootstrap.cc:492] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
@ 0x7fa49a3333ab kudu::Thread::SuperviseThread() at ??:0
@ 0x474190 __tsan_thread_start_func at ??:?
I20251211 08:58:39.321735 17118 tablet_service.cc:1552] Processing DeleteTablet for tablet 81b5f03965634d6db4b2c1737a26bf30 with delete_type TABLET_DATA_DELETED (replaced by tablet 316fbbd2f6e0476e9b9768b954a21fbc) from {username='slave'} at 127.0.0.1:43290
I20251211 08:58:39.322423 17563 tablet_replica.cc:333] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70: stopping tablet replica
I20251211 08:58:39.323115 17563 raft_consensus.cc:2243] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:39.323513 17563 raft_consensus.cc:2272] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:39.325477 17563 ts_tablet_manager.cc:1916] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fa49ba3e6db start_thread at ??:0
I20251211 08:58:39.339216 17558 tablet_bootstrap.cc:654] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:39.342793 17276 tablet_service.cc:1552] Processing DeleteTablet for tablet 81b5f03965634d6db4b2c1737a26bf30 with delete_type TABLET_DATA_DELETED (replaced by tablet 316fbbd2f6e0476e9b9768b954a21fbc) from {username='slave'} at 127.0.0.1:37278
I20251211 08:58:39.343438 17565 tablet_replica.cc:333] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2: stopping tablet replica
I20251211 08:58:39.344039 17565 raft_consensus.cc:2243] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:39.344424 17565 raft_consensus.cc:2272] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:39.346814 17564 ts_tablet_manager.cc:1916] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:39.347316 17565 ts_tablet_manager.cc:1916] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:39.349915 17555 tablet_bootstrap.cc:492] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
I20251211 08:58:39.350281 17555 ts_tablet_manager.cc:1403] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.079s user 0.014s sys 0.008s
I20251211 08:58:39.356108 17555 raft_consensus.cc:359] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.356700 17555 raft_consensus.cc:385] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:39.356964 17555 raft_consensus.cc:740] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:39.362078 17565 ts_tablet_manager.cc:1929] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:39.362416 17565 log.cc:1199] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/81b5f03965634d6db4b2c1737a26bf30
I20251211 08:58:39.363422 17565 ts_tablet_manager.cc:1950] T 81b5f03965634d6db4b2c1737a26bf30 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting consensus metadata
I20251211 08:58:39.365944 17555 consensus_queue.cc:260] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.374931 17276 tablet_service.cc:1505] Processing CreateTablet for tablet 316fbbd2f6e0476e9b9768b954a21fbc (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:39.375950 17276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 316fbbd2f6e0476e9b9768b954a21fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:39.375782 17429 tablet_service.cc:1505] Processing CreateTablet for tablet 316fbbd2f6e0476e9b9768b954a21fbc (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:39.376961 17429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 316fbbd2f6e0476e9b9768b954a21fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:39.379293 17117 tablet_service.cc:1505] Processing CreateTablet for tablet 316fbbd2f6e0476e9b9768b954a21fbc (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:39.387887 17117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 316fbbd2f6e0476e9b9768b954a21fbc. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fa496db671f clone at ??:0
I20251211 08:58:39.397002 17558 tablet_bootstrap.cc:492] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:39.397396 17558 ts_tablet_manager.cc:1403] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.001s
I20251211 08:58:39.397688 17553 tablet_bootstrap.cc:492] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:39.399638 17558 raft_consensus.cc:359] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.401368 17558 raft_consensus.cc:385] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:39.401645 17558 raft_consensus.cc:740] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:39.402370 17553 tablet_bootstrap.cc:654] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:39.403427 17558 consensus_queue.cc:260] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.411166 17558 ts_tablet_manager.cc:1434] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20251211 08:58:39.411710 17555 ts_tablet_manager.cc:1434] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.059s user 0.005s sys 0.000s
I20251211 08:58:39.415500 17564 ts_tablet_manager.cc:1929] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:39.415872 17564 log.cc:1199] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/81b5f03965634d6db4b2c1737a26bf30
I20251211 08:58:39.416473 17564 ts_tablet_manager.cc:1950] T 81b5f03965634d6db4b2c1737a26bf30 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting consensus metadata
I20251211 08:58:39.419309 17563 ts_tablet_manager.cc:1929] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:39.419698 17563 log.cc:1199] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/81b5f03965634d6db4b2c1737a26bf30
I20251211 08:58:39.420342 17563 ts_tablet_manager.cc:1950] T 81b5f03965634d6db4b2c1737a26bf30 P 62329f65aae54e84b396a51dd81f1b70: Deleting consensus metadata
I20251211 08:58:39.426882 17555 tablet_bootstrap.cc:492] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
I20251211 08:58:39.431833 17555 tablet_bootstrap.cc:654] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:39.436522 17553 tablet_bootstrap.cc:492] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:39.436896 17553 ts_tablet_manager.cc:1403] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.003s
W20251211 08:58:39.438453 17554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20251211 08:58:39.439049 17555 tablet_bootstrap.cc:492] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
W20251211 08:58:39.439283 17554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20251211 08:58:39.439401 17555 ts_tablet_manager.cc:1403] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.001s
I20251211 08:58:39.439112 17553 raft_consensus.cc:359] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.439716 17553 raft_consensus.cc:385] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251211 08:58:39.440007 17554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251211 08:58:39.439947 17553 raft_consensus.cc:740] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
I20251211 08:58:39.440447 17553 consensus_queue.cc:260] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.442236 17553 ts_tablet_manager.cc:1434] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251211 08:58:39.445147 17558 tablet_bootstrap.cc:492] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
I20251211 08:58:39.450410 17558 tablet_bootstrap.cc:654] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:39.453249 17555 raft_consensus.cc:359] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.454300 17555 raft_consensus.cc:385] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:39.454649 17555 raft_consensus.cc:740] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:39.455227 17555 consensus_queue.cc:260] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.464284 17555 ts_tablet_manager.cc:1434] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.025s user 0.002s sys 0.002s
I20251211 08:58:39.470980 17561 raft_consensus.cc:493] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:39.471549 17561 raft_consensus.cc:515] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.474449 17561 leader_election.cc:290] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
I20251211 08:58:39.528498 17558 tablet_bootstrap.cc:492] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:39.528959 17558 ts_tablet_manager.cc:1403] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.084s user 0.021s sys 0.018s
I20251211 08:58:39.531195 17558 raft_consensus.cc:359] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.531720 17558 raft_consensus.cc:385] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:39.531951 17558 raft_consensus.cc:740] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:39.532445 17558 consensus_queue.cc:260] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.539084 17558 ts_tablet_manager.cc:1434] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251211 08:58:39.552556 17138 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb7a90c0accf4669a1cdc4591284954f" candidate_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62329f65aae54e84b396a51dd81f1b70" is_pre_election: true
I20251211 08:58:39.553166 17138 raft_consensus.cc:2468] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7467cf4890bc486bbfb9b8f4bea2c2cf in term 0.
I20251211 08:58:39.554311 17382 leader_election.cc:304] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters:
I20251211 08:58:39.555577 17561 raft_consensus.cc:2804] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:39.555868 17561 raft_consensus.cc:493] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:39.556111 17561 raft_consensus.cc:3060] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:39.560868 17296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb7a90c0accf4669a1cdc4591284954f" candidate_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" is_pre_election: true
I20251211 08:58:39.561350 17296 raft_consensus.cc:2468] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7467cf4890bc486bbfb9b8f4bea2c2cf in term 0.
W20251211 08:58:39.563515 17554 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20251211 08:58:39.563843 17554 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
I20251211 08:58:39.563812 17561 raft_consensus.cc:515] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
W20251211 08:58:39.564215 17554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251211 08:58:39.564567 17554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:58:39.564903 17554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251211 08:58:39.565230 17554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251211 08:58:39.567050 17138 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb7a90c0accf4669a1cdc4591284954f" candidate_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62329f65aae54e84b396a51dd81f1b70"
I20251211 08:58:39.567530 17138 raft_consensus.cc:3060] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:39.573698 17561 leader_election.cc:290] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 election: Requested vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
I20251211 08:58:39.574790 17296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb7a90c0accf4669a1cdc4591284954f" candidate_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2"
I20251211 08:58:39.575340 17296 raft_consensus.cc:3060] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:39.575361 17138 raft_consensus.cc:2468] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7467cf4890bc486bbfb9b8f4bea2c2cf in term 1.
I20251211 08:58:39.577337 17382 leader_election.cc:304] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters:
I20251211 08:58:39.578114 17561 raft_consensus.cc:2804] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:39.583269 17561 raft_consensus.cc:697] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 LEADER]: Becoming Leader. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Running, Role: LEADER
I20251211 08:58:39.583380 17296 raft_consensus.cc:2468] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7467cf4890bc486bbfb9b8f4bea2c2cf in term 1.
I20251211 08:58:39.584132 17561 consensus_queue.cc:237] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.633109 17554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:46123
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.60:34701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.146.60
--webserver_port=41887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:58:39.634681 17554 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:58:39.637566 17554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:58:39.652340 17628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:39.751956 17562 raft_consensus.cc:493] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:39.752604 17562 raft_consensus.cc:515] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.756348 17562 leader_election.cc:290] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:39.758323 17138 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "316fbbd2f6e0476e9b9768b954a21fbc" candidate_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62329f65aae54e84b396a51dd81f1b70" is_pre_election: true
I20251211 08:58:39.759014 17138 raft_consensus.cc:2468] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8a2e7dba69c4c80bb3672eeaf63e1c2 in term 0.
I20251211 08:58:39.759423 17449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "316fbbd2f6e0476e9b9768b954a21fbc" candidate_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" is_pre_election: true
I20251211 08:58:39.760195 17449 raft_consensus.cc:2468] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8a2e7dba69c4c80bb3672eeaf63e1c2 in term 0.
I20251211 08:58:39.761967 17229 leader_election.cc:304] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2; no voters:
I20251211 08:58:39.763620 17562 raft_consensus.cc:2804] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:39.764040 17562 raft_consensus.cc:493] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:39.764430 17562 raft_consensus.cc:3060] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:39.773170 17562 raft_consensus.cc:515] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:39.776504 17138 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "316fbbd2f6e0476e9b9768b954a21fbc" candidate_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62329f65aae54e84b396a51dd81f1b70"
I20251211 08:58:39.777016 17138 raft_consensus.cc:3060] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:39.777897 17449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "316fbbd2f6e0476e9b9768b954a21fbc" candidate_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf"
I20251211 08:58:39.778614 17449 raft_consensus.cc:3060] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:39.784744 17138 raft_consensus.cc:2468] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8a2e7dba69c4c80bb3672eeaf63e1c2 in term 1.
I20251211 08:58:39.786134 17229 leader_election.cc:304] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2; no voters:
I20251211 08:58:39.793840 17562 leader_election.cc:290] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 election: Requested vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:39.795401 17449 raft_consensus.cc:2468] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8a2e7dba69c4c80bb3672eeaf63e1c2 in term 1.
I20251211 08:58:39.796512 17562 raft_consensus.cc:2804] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:39.797642 17562 raft_consensus.cc:697] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 LEADER]: Becoming Leader. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Running, Role: LEADER
I20251211 08:58:39.800446 17562 consensus_queue.cc:237] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
W20251211 08:58:39.834074 17230 connection.cc:537] server connection from 127.0.0.1:37278 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:39.834725 17504 connection.cc:537] client connection to 127.13.146.61:36035 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:39.835161 17070 connection.cc:537] client connection to 127.13.146.61:36035 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:39.835920 17073 connection.cc:537] server connection from 127.0.0.1:43290 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:39.837018 17504 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.13.146.61:36035)
W20251211 08:58:39.838171 17504 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.146.61:36035)
W20251211 08:58:39.838995 17382 connection.cc:537] server connection from 127.0.0.1:52092 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:39.839516 17382 connection.cc:537] client connection to 127.13.146.61:36035 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:39.844220 17229 connection.cc:537] client connection to 127.13.146.61:36035 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:39.854202 17504 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.13.146.61:36035)
W20251211 08:58:39.857606 17504 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.13.146.61:36035)
I20251211 08:58:39.871500 17637 raft_consensus.cc:493] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c3112fc6f4ae4541a2af2405a97c707b)
I20251211 08:58:39.871924 17637 raft_consensus.cc:515] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:39.886744 17637 leader_election.cc:290] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:58:39.894833 16763 leader_election.cc:336] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 08:58:39.898280 16761 leader_election.cc:336] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035): Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:39.898658 16761 leader_election.cc:304] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800; no voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:39.899276 17637 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251211 08:58:39.983700 17592 consensus_queue.cc:1048] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:58:40.045312 17592 consensus_queue.cc:1048] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:40.367069 17561 consensus_queue.cc:1048] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:58:40.512609 17655 raft_consensus.cc:493] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c3112fc6f4ae4541a2af2405a97c707b)
I20251211 08:58:40.519289 17652 consensus_queue.cc:1048] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251211 08:58:40.540135 17655 raft_consensus.cc:515] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:40.542500 17655 leader_election.cc:290] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:58:40.570499 16763 leader_election.cc:336] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 08:58:40.570984 16761 leader_election.cc:336] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035): Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:40.571352 16761 leader_election.cc:304] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800; no voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:40.572295 17655 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251211 08:58:40.601923 17562 consensus_queue.cc:1048] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:40.656988 17661 consensus_queue.cc:1048] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251211 08:58:41.052317 17627 debug-util.cc:398] Leaking SignalData structure 0x7b0800028120 after lost signal to thread 17554
W20251211 08:58:39.666509 17629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:41.136541 17627 kernel_stack_watchdog.cc:198] Thread 17554 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 389ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20251211 08:58:41.137831 17554 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.477s user 0.537s sys 0.854s
W20251211 08:58:41.138283 17554 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.477s user 0.537s sys 0.854s
I20251211 08:58:41.150717 17554 server_base.cc:1047] running on GCE node
W20251211 08:58:41.151345 17668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:41.152493 17554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:58:41.155862 17554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:58:41.157341 17554 hybrid_clock.cc:648] HybridClock initialized: now 1765443521157280 us; error 49 us; skew 500 ppm
I20251211 08:58:41.158229 17554 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:58:41.167070 17554 webserver.cc:492] Webserver started at http://127.13.146.60:41887/ using document root <none> and password file <none>
I20251211 08:58:41.168025 17554 fs_manager.cc:362] Metadata directory not provided
I20251211 08:58:41.168256 17554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:58:41.175828 17554 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20251211 08:58:41.180944 17675 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:41.182487 17554 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20251211 08:58:41.182893 17554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal
uuid: "b8dd1b89d5ad4e529459606ce6f85ab8"
format_stamp: "Formatted at 2025-12-11 08:58:28 on dist-test-slave-0dfs"
I20251211 08:58:41.184902 17554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:58:41.245070 17554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:58:41.246687 17554 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:58:41.247282 17554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:58:41.327580 17554 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.60:34701
I20251211 08:58:41.327602 17729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.60:34701 every 8 connection(s)
I20251211 08:58:41.330408 17554 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-2/data/info.pb
I20251211 08:58:41.335245 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 17554
I20251211 08:58:41.344978 17730 sys_catalog.cc:263] Verifying existing consensus state
I20251211 08:58:41.353830 17730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Bootstrap starting.
I20251211 08:58:41.380722 17730 log.cc:826] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Log is configured to *not* fsync() on all Append() calls
I20251211 08:58:41.409500 17730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251211 08:58:41.410589 17730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Bootstrap complete.
I20251211 08:58:41.417220 17494 heartbeater.cc:344] Connected to a master server at 127.13.146.60:34701
I20251211 08:58:41.417665 17494 heartbeater.cc:499] Master 127.13.146.60:34701 was elected leader, sending a full tablet report...
I20251211 08:58:41.432616 17730 raft_consensus.cc:359] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:41.434922 17730 raft_consensus.cc:740] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8dd1b89d5ad4e529459606ce6f85ab8, State: Initialized, Role: FOLLOWER
I20251211 08:58:41.435640 17730 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:41.439074 17737 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:41.439875 17737 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:41.440968 17730 sys_catalog.cc:565] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: configured and running, proceeding with master startup.
I20251211 08:58:41.461684 17341 heartbeater.cc:344] Connected to a master server at 127.13.146.60:34701
I20251211 08:58:41.486009 17759 raft_consensus.cc:493] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c3112fc6f4ae4541a2af2405a97c707b)
I20251211 08:58:41.486438 17759 raft_consensus.cc:515] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:41.488175 17759 leader_election.cc:290] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:41.502096 17755 catalog_manager.cc:1269] Loaded cluster ID: c258c1a306e24c4197d31ee43870f1fb
I20251211 08:58:41.502457 17755 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: loading cluster ID for follower catalog manager: success
W20251211 08:58:41.507826 16761 leader_election.cc:336] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035): Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:41.508345 17704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 7 candidate_status { last_received { term: 6 index: 27 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:41.509423 17704 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133269af8ce44b579ebbb7805bc58800 in term 5.
I20251211 08:58:41.510847 16763 leader_election.cc:304] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8; no voters: c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:41.511404 17755 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: acquiring CA information for follower catalog manager: success
I20251211 08:58:41.511766 17759 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251211 08:58:41.512131 17759 raft_consensus.cc:493] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c3112fc6f4ae4541a2af2405a97c707b)
I20251211 08:58:41.512475 17759 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 6 FOLLOWER]: Advancing to term 7
I20251211 08:58:41.516680 17755 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251211 08:58:41.519569 17759 raft_consensus.cc:515] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:41.524021 17704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133269af8ce44b579ebbb7805bc58800" candidate_term: 7 candidate_status { last_received { term: 6 index: 27 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8"
I20251211 08:58:41.524618 17704 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 5 FOLLOWER]: Advancing to term 7
W20251211 08:58:41.525820 16761 leader_election.cc:336] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035): Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:41.523074 17759 leader_election.cc:290] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 election: Requested vote from peers c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:41.544396 17183 heartbeater.cc:344] Connected to a master server at 127.13.146.60:34701
I20251211 08:58:41.514834 17554 master_runner.cc:186] Error getting master registration for 127.13.146.61:36035: Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111),
I20251211 08:58:41.545156 17554 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:58:41.552688 17704 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133269af8ce44b579ebbb7805bc58800 in term 7.
I20251211 08:58:41.554476 16763 leader_election.cc:304] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8; no voters: c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:41.555385 17759 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 7 FOLLOWER]: Leader election won for term 7
I20251211 08:58:41.556664 17759 raft_consensus.cc:697] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [term 7 LEADER]: Becoming Leader. State: Replica: 133269af8ce44b579ebbb7805bc58800, State: Running, Role: LEADER
I20251211 08:58:41.557464 17759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 6.27, Last appended by leader: 27, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:41.562664 17765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133269af8ce44b579ebbb7805bc58800. Latest consensus state: current_term: 7 leader_uuid: "133269af8ce44b579ebbb7805bc58800" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:41.563453 17765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: This master's current role is: LEADER
I20251211 08:58:41.567294 17694 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" instance_seqno: 1765443513006607) as {username='slave'} at 127.13.146.1:47533; Asking this server to re-register.
I20251211 08:58:41.569397 17183 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:41.572707 17767 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251211 08:58:41.574064 17694 ts_manager.cc:194] Registered new tserver with Master: 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231)
I20251211 08:58:41.580130 17767 catalog_manager.cc:679] Loaded metadata for table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]
I20251211 08:58:41.581555 17767 catalog_manager.cc:679] Loaded metadata for table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]
I20251211 08:58:41.582693 17767 catalog_manager.cc:679] Loaded metadata for table default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]
I20251211 08:58:41.588474 17767 tablet_loader.cc:96] loaded metadata for tablet 316fbbd2f6e0476e9b9768b954a21fbc (table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299])
I20251211 08:58:41.589726 17767 tablet_loader.cc:96] loaded metadata for tablet 61ae46a42d1e46e3a4754213c2d19259 (table default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283])
I20251211 08:58:41.591643 17767 tablet_loader.cc:96] loaded metadata for tablet cb7a90c0accf4669a1cdc4591284954f (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4])
I20251211 08:58:41.592983 17767 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251211 08:58:41.597920 17767 catalog_manager.cc:1269] Loaded cluster ID: c258c1a306e24c4197d31ee43870f1fb
I20251211 08:58:41.598160 17767 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251211 08:58:41.603543 17767 catalog_manager.cc:1514] Loading token signing keys...
I20251211 08:58:41.608341 17767 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: Loaded TSK: 0
I20251211 08:58:41.609228 17767 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251211 08:58:41.614537 17767 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251211 08:58:41.619877 17767 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251211 08:58:41.633317 17704 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Refusing update from remote peer 133269af8ce44b579ebbb7805bc58800: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 29. (index mismatch)
I20251211 08:58:41.634758 17759 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 6, Time since last communication: 0.000s
W20251211 08:58:41.636394 16761 consensus_peers.cc:597] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 -> Peer c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035): Couldn't send request to peer c3112fc6f4ae4541a2af2405a97c707b. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:41.680800 17737 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133269af8ce44b579ebbb7805bc58800. Latest consensus state: current_term: 7 leader_uuid: "133269af8ce44b579ebbb7805bc58800" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:41.681528 17737 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:41.681787 17768 mvcc.cc:204] Tried to move back new op lower bound from 7231256664594362368 to 7231256664308768768. Current Snapshot: MvccSnapshot[applied={T|T < 7231256664594362368}]
I20251211 08:58:41.683998 17771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "133269af8ce44b579ebbb7805bc58800" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:41.684463 17771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:41.684835 17771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "133269af8ce44b579ebbb7805bc58800" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:41.685318 17771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:41.689977 16776 master_service.cc:954] ReplaceTablet: received request to replace tablet 61ae46a42d1e46e3a4754213c2d19259 from {username='slave'} at 127.0.0.1:33204
I20251211 08:58:41.691295 16777 master_service.cc:954] ReplaceTablet: received request to replace tablet 61ae46a42d1e46e3a4754213c2d19259 from {username='slave'} at 127.0.0.1:33204
I20251211 08:58:41.702250 17759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "133269af8ce44b579ebbb7805bc58800" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:41.702981 17759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800 [sys.catalog]: This master's current role is: LEADER
I20251211 08:58:41.722414 17769 mvcc.cc:204] Tried to move back new op lower bound from 7231256664594362368 to 7231256648114573312. Current Snapshot: MvccSnapshot[applied={T|T < 7231256664594362368}]
F20251211 08:58:41.727684 17043 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1765443517, eventType=CREATE_TABLE, dbName=default, tableName=table_1bf93acd2af64f98b4cda7ce375e702e, message={"server":"","servicePrincipal":"","db":"default","table":"table_1bf93acd2af64f98b4cda7ce375e702e","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_1bf93acd2af64f98b4cda7ce375e702e\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765443517},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/warehouse/table_1bf93acd2af64f98b4cda7ce375e702e\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_1bf93acd2af64f98b4cda7ce375e702e\",\"kudu.table_id\":\"86b322f346e0458daf6c4f9dd962de20\",\"transient_lastDdlTime\":\"1765443517\",\"comment\":\"\",\"kudu.master_addresses\":\"127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"c258c1a306e24c4197d31ee43870f1fb\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765443517,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765443517, eventType=CREATE_TABLE, dbName=default, tableName=table_d36118f54fec423badd3d2b59d64b05c, message={"server":"","servicePrincipal":"","db":"default","table":"table_d36118f54fec423badd3d2b59d64b05c","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d36118f54fec423badd3d2b59d64b05c\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765443517},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/warehouse/table_d36118f54fec423badd3d2b59d64b05c\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d36118f54fec423badd3d2b59d64b05c\",\"kudu.table_id\":\"53d392cd36244b498c2740a372a1daf4\",\"transient_lastDdlTime\":\"1765443517\",\"comment\":\"\",\"kudu.master_addresses\":\"127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"c258c1a306e24c4197d31ee43870f1fb\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765443517,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251211 08:58:41.718088 16770 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33204:
table { table_name: "default.Table_d36118f54fec423badd3d2b59d64b05c" } new_table_name: "default.Table_c00e0c8e65c24aac8676a17546efe6eb" modify_external_catalogs: true
I20251211 08:58:41.706893 16771 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33204:
table { table_name: "default.Table_12ea97bf52084b09955460036752d6e4" } modify_external_catalogs: true
I20251211 08:58:41.712185 16775 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33204:
name: "default.Table_7bc946992a8e44a0987ff88263b4c255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:41.717303 16769 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33204:
table { table_name: "default.Table_da60da9af7664a10a99d2fa227af8235" } new_table_name: "default.Table_59e9546b87cc408f961a8f029510dc53" modify_external_catalogs: true
I20251211 08:58:41.704564 16772 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33204:
name: "default.Table_eff061b04cb14ce88f2a8555db7504ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:41.703462 16774 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33204:
name: "default.Table_9088340c00c94ec8bb9312a333dec04d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:41.711580 16773 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33204:
name: "default.Table_097cc85012564d15aa8db749266d3068"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:41.747503 16771 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33204:
table { table_name: "default.Table_12ea97bf52084b09955460036752d6e4" } modify_external_catalogs: true
I20251211 08:58:41.759999 16777 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: Sending DeleteTablet for 3 replicas of tablet 61ae46a42d1e46e3a4754213c2d19259
I20251211 08:58:41.763645 16777 catalog_manager.cc:6724] ReplaceTablet: tablet 61ae46a42d1e46e3a4754213c2d19259 deleted and replaced by tablet 128b128c966b4dcfa6241b96e3ae61b1
@ 0x7fb3680c5950 google::LogMessage::Flush() at ??:0
@ 0x7fb3680c56bf google::LogMessage::~LogMessage() at ??:0
@ 0x7fb37315826c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251211 08:58:41.839732 17117 tablet_service.cc:1552] Processing DeleteTablet for tablet 61ae46a42d1e46e3a4754213c2d19259 with delete_type TABLET_DATA_DELETED (replaced by tablet 128b128c966b4dcfa6241b96e3ae61b1) from {username='slave'} at 127.0.0.1:43306
I20251211 08:58:41.843452 17118 tablet_service.cc:1505] Processing CreateTablet for tablet 128b128c966b4dcfa6241b96e3ae61b1 (DEFAULT_TABLE table=default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:41.844731 17118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 128b128c966b4dcfa6241b96e3ae61b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:41.847843 17784 tablet_replica.cc:333] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70: stopping tablet replica
I20251211 08:58:41.848711 17276 tablet_service.cc:1552] Processing DeleteTablet for tablet 61ae46a42d1e46e3a4754213c2d19259 with delete_type TABLET_DATA_DELETED (replaced by tablet 128b128c966b4dcfa6241b96e3ae61b1) from {username='slave'} at 127.0.0.1:37282
I20251211 08:58:41.848661 17784 raft_consensus.cc:2243] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:41.849874 17784 raft_consensus.cc:2272] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:41.851936 17275 tablet_service.cc:1505] Processing CreateTablet for tablet 128b128c966b4dcfa6241b96e3ae61b1 (DEFAULT_TABLE table=default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:41.853013 17275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 128b128c966b4dcfa6241b96e3ae61b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:41.857214 17429 tablet_service.cc:1552] Processing DeleteTablet for tablet 61ae46a42d1e46e3a4754213c2d19259 with delete_type TABLET_DATA_DELETED (replaced by tablet 128b128c966b4dcfa6241b96e3ae61b1) from {username='slave'} at 127.0.0.1:52100
I20251211 08:58:41.861088 17428 tablet_service.cc:1505] Processing CreateTablet for tablet 128b128c966b4dcfa6241b96e3ae61b1 (DEFAULT_TABLE table=default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:41.862537 17428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 128b128c966b4dcfa6241b96e3ae61b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:41.864012 17784 ts_tablet_manager.cc:1916] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:41.866436 17785 tablet_replica.cc:333] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2: stopping tablet replica
I20251211 08:58:41.867254 17785 raft_consensus.cc:2243] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:41.867781 17785 raft_consensus.cc:2272] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:41.869657 17785 ts_tablet_manager.cc:1916] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:41.875730 17786 tablet_replica.cc:333] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf: stopping tablet replica
I20251211 08:58:41.876580 17786 raft_consensus.cc:2243] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:41.877106 17786 raft_consensus.cc:2272] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fb373157a2c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251211 08:58:41.880868 17786 ts_tablet_manager.cc:1916] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:41.885995 17787 tablet_bootstrap.cc:492] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:41.899461 17790 tablet_bootstrap.cc:492] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
I20251211 08:58:41.899811 17788 tablet_bootstrap.cc:492] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
I20251211 08:58:41.904893 17787 tablet_bootstrap.cc:654] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:41.909375 17784 ts_tablet_manager.cc:1929] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:41.909801 17784 log.cc:1199] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/61ae46a42d1e46e3a4754213c2d19259
I20251211 08:58:41.912446 17788 tablet_bootstrap.cc:654] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
@ 0x7fb37315baa2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251211 08:58:41.916184 17786 ts_tablet_manager.cc:1929] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:41.916541 17786 log.cc:1199] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/61ae46a42d1e46e3a4754213c2d19259
I20251211 08:58:41.917236 17786 ts_tablet_manager.cc:1950] T 61ae46a42d1e46e3a4754213c2d19259 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting consensus metadata
I20251211 08:58:41.910606 17784 ts_tablet_manager.cc:1950] T 61ae46a42d1e46e3a4754213c2d19259 P 62329f65aae54e84b396a51dd81f1b70: Deleting consensus metadata
I20251211 08:58:41.919888 17785 ts_tablet_manager.cc:1929] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:41.920265 17785 log.cc:1199] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/61ae46a42d1e46e3a4754213c2d19259
I20251211 08:58:41.921484 16761 catalog_manager.cc:4985] TS 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): tablet 61ae46a42d1e46e3a4754213c2d19259 (table default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]) successfully deleted
I20251211 08:58:41.922827 16762 catalog_manager.cc:4985] TS 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): tablet 61ae46a42d1e46e3a4754213c2d19259 (table default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]) successfully deleted
I20251211 08:58:41.923182 17494 heartbeater.cc:499] Master 127.13.146.60:34701 was elected leader, sending a full tablet report...
I20251211 08:58:41.925874 17785 ts_tablet_manager.cc:1950] T 61ae46a42d1e46e3a4754213c2d19259 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting consensus metadata
I20251211 08:58:41.927953 17694 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" instance_seqno: 1765443517190554) as {username='slave'} at 127.13.146.3:57359; Asking this server to re-register.
I20251211 08:58:41.928372 16761 catalog_manager.cc:4985] TS e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): tablet 61ae46a42d1e46e3a4754213c2d19259 (table default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]) successfully deleted
I20251211 08:58:41.929373 17788 tablet_bootstrap.cc:492] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:41.929523 17494 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:41.929883 17788 ts_tablet_manager.cc:1403] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.000s
I20251211 08:58:41.930629 17790 tablet_bootstrap.cc:654] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:41.933082 17694 ts_manager.cc:194] Registered new tserver with Master: 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:41.932703 17788 raft_consensus.cc:359] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:41.933517 17788 raft_consensus.cc:385] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:41.933776 17788 raft_consensus.cc:740] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:41.934543 17788 consensus_queue.cc:260] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
@ 0x7fb37315ba5a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20251211 08:58:41.970432 17341 heartbeater.cc:344] Connected to a master server at 127.13.146.60:34701
@ 0x7fb37315b9ea std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20251211 08:58:41.976580 17694 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" instance_seqno: 1765443515305506) as {username='slave'} at 127.13.146.2:37165; Asking this server to re-register.
I20251211 08:58:41.979084 17341 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:41.984036 17694 ts_manager.cc:194] Registered new tserver with Master: e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
I20251211 08:58:41.969117 17790 tablet_bootstrap.cc:492] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
I20251211 08:58:41.990191 17790 ts_tablet_manager.cc:1403] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.004s
I20251211 08:58:41.993638 17790 raft_consensus.cc:359] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:41.994390 17790 raft_consensus.cc:385] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:41.994537 16776 master_service.cc:954] ReplaceTablet: received request to replace tablet cb7a90c0accf4669a1cdc4591284954f from {username='slave'} at 127.0.0.1:33204
I20251211 08:58:41.997156 17787 tablet_bootstrap.cc:492] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:41.997820 17787 ts_tablet_manager.cc:1403] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.112s user 0.016s sys 0.000s
I20251211 08:58:41.999493 16777 master_service.cc:954] ReplaceTablet: received request to replace tablet 128b128c966b4dcfa6241b96e3ae61b1 from {username='slave'} at 127.0.0.1:33204
I20251211 08:58:42.002250 17787 raft_consensus.cc:359] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:42.003151 17787 raft_consensus.cc:385] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:42.003471 17787 raft_consensus.cc:740] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
@ 0x7fb37315b9b2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20251211 08:58:42.006314 17788 ts_tablet_manager.cc:1434] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.076s user 0.035s sys 0.041s
I20251211 08:58:42.022395 17787 consensus_queue.cc:260] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:42.025949 17343 heartbeater.cc:499] Master 127.13.146.62:43161 was elected leader, sending a full tablet report...
I20251211 08:58:42.030100 16776 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: Sending DeleteTablet for 3 replicas of tablet cb7a90c0accf4669a1cdc4591284954f
I20251211 08:58:42.030396 17790 raft_consensus.cc:740] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:42.031423 16776 catalog_manager.cc:6724] ReplaceTablet: tablet cb7a90c0accf4669a1cdc4591284954f deleted and replaced by tablet 1ca4a92a351b4a4fa77ef805bfcdb451
@ 0x7fb37315acae std::__1::__function::__func<>::operator()() at ??:0
I20251211 08:58:42.032052 17275 tablet_service.cc:1552] Processing DeleteTablet for tablet cb7a90c0accf4669a1cdc4591284954f with delete_type TABLET_DATA_DELETED (replaced by tablet 1ca4a92a351b4a4fa77ef805bfcdb451) from {username='slave'} at 127.0.0.1:37282
I20251211 08:58:42.032533 17118 tablet_service.cc:1552] Processing DeleteTablet for tablet cb7a90c0accf4669a1cdc4591284954f with delete_type TABLET_DATA_DELETED (replaced by tablet 1ca4a92a351b4a4fa77ef805bfcdb451) from {username='slave'} at 127.0.0.1:43306
I20251211 08:58:42.031121 17790 consensus_queue.cc:260] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:42.032792 17785 tablet_replica.cc:333] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2: stopping tablet replica
I20251211 08:58:42.033435 17785 raft_consensus.cc:2243] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:42.033589 17428 tablet_service.cc:1552] Processing DeleteTablet for tablet cb7a90c0accf4669a1cdc4591284954f with delete_type TABLET_DATA_DELETED (replaced by tablet 1ca4a92a351b4a4fa77ef805bfcdb451) from {username='slave'} at 127.0.0.1:52100
I20251211 08:58:42.033875 17785 raft_consensus.cc:2272] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:42.034147 17784 tablet_replica.cc:333] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70: stopping tablet replica
I20251211 08:58:42.034288 17786 tablet_replica.cc:333] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf: stopping tablet replica
I20251211 08:58:42.034843 17784 raft_consensus.cc:2243] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:42.034994 17786 raft_consensus.cc:2243] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:42.035290 17784 raft_consensus.cc:2272] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:42.035457 17787 ts_tablet_manager.cc:1434] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s
I20251211 08:58:42.035631 17785 ts_tablet_manager.cc:1916] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:42.035959 17786 raft_consensus.cc:2272] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:42.036932 17784 ts_tablet_manager.cc:1916] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:42.038009 17786 ts_tablet_manager.cc:1916] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x5aa2b5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5aa0ea std::__1::function<>::operator()() at ??:?
I20251211 08:58:42.062829 16777 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 133269af8ce44b579ebbb7805bc58800: Sending DeleteTablet for 3 replicas of tablet 128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:58:42.064556 17428 tablet_service.cc:1552] Processing DeleteTablet for tablet 128b128c966b4dcfa6241b96e3ae61b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 0ee60b7cb9094553af8be206d4e38c22) from {username='slave'} at 127.0.0.1:52100
I20251211 08:58:42.065246 17118 tablet_service.cc:1552] Processing DeleteTablet for tablet 128b128c966b4dcfa6241b96e3ae61b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 0ee60b7cb9094553af8be206d4e38c22) from {username='slave'} at 127.0.0.1:43306
I20251211 08:58:42.067167 17275 tablet_service.cc:1552] Processing DeleteTablet for tablet 128b128c966b4dcfa6241b96e3ae61b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 0ee60b7cb9094553af8be206d4e38c22) from {username='slave'} at 127.0.0.1:37282
I20251211 08:58:42.067821 16777 catalog_manager.cc:6724] ReplaceTablet: tablet 128b128c966b4dcfa6241b96e3ae61b1 deleted and replaced by tablet 0ee60b7cb9094553af8be206d4e38c22
@ 0x7fb36a2e73ab kudu::Thread::SuperviseThread() at ??:0
@ 0x474190 __tsan_thread_start_func at ??:?
I20251211 08:58:42.097707 17185 heartbeater.cc:499] Master 127.13.146.62:43161 was elected leader, sending a full tablet report...
I20251211 08:58:42.099932 17786 ts_tablet_manager.cc:1929] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:42.100342 17786 log.cc:1199] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/cb7a90c0accf4669a1cdc4591284954f
I20251211 08:58:42.101022 17786 ts_tablet_manager.cc:1950] T cb7a90c0accf4669a1cdc4591284954f P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting consensus metadata
W20251211 08:58:42.101994 16768 catalog_manager.cc:5487] Ignoring report from unknown tablet 128b128c966b4dcfa6241b96e3ae61b1
W20251211 08:58:42.102434 16768 catalog_manager.cc:5487] Ignoring report from unknown tablet cb7a90c0accf4669a1cdc4591284954f
I20251211 08:58:42.103277 16762 catalog_manager.cc:4985] TS 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): tablet cb7a90c0accf4669a1cdc4591284954f (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]) successfully deleted
W20251211 08:58:42.104331 16762 catalog_manager.cc:4965] TS 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): delete failed for tablet 128b128c966b4dcfa6241b96e3ae61b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 128b128c966b4dcfa6241b96e3ae61b1 already in progress: creating tablet
@ 0x7fb36b9f26db start_thread at ??:0
I20251211 08:58:42.123780 17496 heartbeater.cc:499] Master 127.13.146.62:43161 was elected leader, sending a full tablet report...
I20251211 08:58:42.124836 17790 ts_tablet_manager.cc:1434] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.134s user 0.026s sys 0.042s
W20251211 08:58:42.127976 16777 catalog_manager.cc:5487] Ignoring report from unknown tablet 128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:58:42.133888 16776 catalog_manager.cc:5654] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } health_report { overall_health: HEALTHY } } }
I20251211 08:58:42.137045 17275 tablet_service.cc:1505] Processing CreateTablet for tablet 0ee60b7cb9094553af8be206d4e38c22 (DEFAULT_TABLE table=default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:42.137495 17785 ts_tablet_manager.cc:1929] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:42.137876 17785 log.cc:1199] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/cb7a90c0accf4669a1cdc4591284954f
I20251211 08:58:42.138110 17275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee60b7cb9094553af8be206d4e38c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:42.138000 17428 tablet_service.cc:1505] Processing CreateTablet for tablet 0ee60b7cb9094553af8be206d4e38c22 (DEFAULT_TABLE table=default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:42.139043 17428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee60b7cb9094553af8be206d4e38c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:42.142692 17117 tablet_service.cc:1505] Processing CreateTablet for tablet 1ca4a92a351b4a4fa77ef805bfcdb451 (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:42.142935 17118 tablet_service.cc:1505] Processing CreateTablet for tablet 0ee60b7cb9094553af8be206d4e38c22 (DEFAULT_TABLE table=default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:42.143656 17117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca4a92a351b4a4fa77ef805bfcdb451. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:42.144205 17118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee60b7cb9094553af8be206d4e38c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:42.139797 17784 ts_tablet_manager.cc:1929] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:42.154587 17784 log.cc:1199] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/cb7a90c0accf4669a1cdc4591284954f
I20251211 08:58:42.156163 17784 ts_tablet_manager.cc:1950] T cb7a90c0accf4669a1cdc4591284954f P 62329f65aae54e84b396a51dd81f1b70: Deleting consensus metadata
I20251211 08:58:42.138603 17785 ts_tablet_manager.cc:1950] T cb7a90c0accf4669a1cdc4591284954f P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting consensus metadata
I20251211 08:58:42.159571 17276 tablet_service.cc:1505] Processing CreateTablet for tablet 1ca4a92a351b4a4fa77ef805bfcdb451 (DEFAULT_TABLE table=default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:42.160619 17276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca4a92a351b4a4fa77ef805bfcdb451. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb366d6a71f clone at ??:0
I20251211 08:58:42.162125 17790 tablet_bootstrap.cc:492] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
I20251211 08:58:42.169180 17790 tablet_bootstrap.cc:654] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:42.178133 17787 tablet_bootstrap.cc:492] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:42.178644 17788 tablet_bootstrap.cc:492] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
I20251211 08:58:42.184580 17787 tablet_bootstrap.cc:654] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:42.184381 17788 tablet_bootstrap.cc:654] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:42.187108 17785 tablet_replica.cc:333] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2: stopping tablet replica
I20251211 08:58:42.188125 17784 tablet_replica.cc:333] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70: stopping tablet replica
I20251211 08:58:42.188838 17790 tablet_bootstrap.cc:492] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
I20251211 08:58:42.188903 17784 raft_consensus.cc:2243] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:42.189276 17790 ts_tablet_manager.cc:1403] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.007s
I20251211 08:58:42.189357 17784 raft_consensus.cc:2272] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:42.190234 17785 raft_consensus.cc:2243] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:42.190737 17785 raft_consensus.cc:2272] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:42.194057 17785 ts_tablet_manager.cc:1916] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:42.195946 17784 ts_tablet_manager.cc:1916] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:42.201486 17788 tablet_bootstrap.cc:492] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:42.201895 17788 ts_tablet_manager.cc:1403] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.002s
I20251211 08:58:42.204367 17788 raft_consensus.cc:359] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.205006 17788 raft_consensus.cc:385] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:42.205250 17788 raft_consensus.cc:740] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:42.205921 17788 consensus_queue.cc:260] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.208669 17787 tablet_bootstrap.cc:492] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:42.209110 17787 ts_tablet_manager.cc:1403] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20251211 08:58:42.216630 17790 raft_consensus.cc:359] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.217315 17784 ts_tablet_manager.cc:1929] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:42.217782 17784 log.cc:1199] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:58:42.218233 17788 ts_tablet_manager.cc:1434] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20251211 08:58:42.218997 17790 raft_consensus.cc:385] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:42.219285 17790 raft_consensus.cc:740] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:42.219923 17790 consensus_queue.cc:260] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.222169 17790 ts_tablet_manager.cc:1434] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.033s user 0.001s sys 0.005s
I20251211 08:58:42.233466 17785 ts_tablet_manager.cc:1929] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251211 08:58:42.233906 17785 log.cc:1199] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:58:42.218585 17784 ts_tablet_manager.cc:1950] T 128b128c966b4dcfa6241b96e3ae61b1 P 62329f65aae54e84b396a51dd81f1b70: Deleting consensus metadata
I20251211 08:58:42.234611 17785 ts_tablet_manager.cc:1950] T 128b128c966b4dcfa6241b96e3ae61b1 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting consensus metadata
I20251211 08:58:42.234072 17787 raft_consensus.cc:359] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.237192 17787 raft_consensus.cc:385] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:42.237568 17787 raft_consensus.cc:740] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
I20251211 08:58:42.238319 17787 consensus_queue.cc:260] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.239413 17788 tablet_bootstrap.cc:492] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
I20251211 08:58:42.245308 17788 tablet_bootstrap.cc:654] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:42.247097 17787 ts_tablet_manager.cc:1434] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20251211 08:58:42.250568 17787 tablet_bootstrap.cc:492] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:42.260005 17787 tablet_bootstrap.cc:654] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:42.266894 17788 tablet_bootstrap.cc:492] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:42.267320 17787 tablet_bootstrap.cc:492] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:42.267807 17787 ts_tablet_manager.cc:1403] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20251211 08:58:42.268290 17788 ts_tablet_manager.cc:1403] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.004s
I20251211 08:58:42.273219 17788 raft_consensus.cc:359] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.273999 17787 raft_consensus.cc:359] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.274739 17787 raft_consensus.cc:385] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:42.275128 17788 raft_consensus.cc:385] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:42.275475 17787 raft_consensus.cc:740] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
I20251211 08:58:42.275871 17788 raft_consensus.cc:740] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:42.276901 17787 consensus_queue.cc:260] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.278025 17788 consensus_queue.cc:260] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.286937 17787 ts_tablet_manager.cc:1434] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.018s user 0.001s sys 0.006s
I20251211 08:58:42.292933 17788 ts_tablet_manager.cc:1434] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.022s user 0.001s sys 0.004s
I20251211 08:58:42.373071 17742 raft_consensus.cc:493] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:42.373572 17742 raft_consensus.cc:515] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.375707 17742 leader_election.cc:290] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:42.378237 17138 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ca4a92a351b4a4fa77ef805bfcdb451" candidate_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62329f65aae54e84b396a51dd81f1b70" is_pre_election: true
I20251211 08:58:42.378890 17138 raft_consensus.cc:2468] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8a2e7dba69c4c80bb3672eeaf63e1c2 in term 0.
W20251211 08:58:42.381073 17230 leader_election.cc:343] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): Not found: Tablet not found: 1ca4a92a351b4a4fa77ef805bfcdb451
I20251211 08:58:42.382092 17229 leader_election.cc:304] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2; no voters: 7467cf4890bc486bbfb9b8f4bea2c2cf
I20251211 08:58:42.383082 17742 raft_consensus.cc:2804] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:42.383410 17742 raft_consensus.cc:493] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:42.383787 17742 raft_consensus.cc:3060] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:42.390220 17742 raft_consensus.cc:515] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.399195 17794 raft_consensus.cc:493] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:42.399255 17138 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ca4a92a351b4a4fa77ef805bfcdb451" candidate_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62329f65aae54e84b396a51dd81f1b70"
I20251211 08:58:42.399871 17138 raft_consensus.cc:3060] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:42.399765 17794 raft_consensus.cc:515] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.401671 17794 leader_election.cc:290] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:42.402534 17296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ee60b7cb9094553af8be206d4e38c22" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" is_pre_election: true
I20251211 08:58:42.403123 17296 raft_consensus.cc:2468] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62329f65aae54e84b396a51dd81f1b70 in term 0.
I20251211 08:58:42.404116 17449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ee60b7cb9094553af8be206d4e38c22" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" is_pre_election: true
I20251211 08:58:42.404462 17070 leader_election.cc:304] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2; no voters:
I20251211 08:58:42.404734 17449 raft_consensus.cc:2468] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62329f65aae54e84b396a51dd81f1b70 in term 0.
I20251211 08:58:42.406484 17794 raft_consensus.cc:2804] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:42.406810 17794 raft_consensus.cc:493] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:42.407110 17794 raft_consensus.cc:3060] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:42.410086 17138 raft_consensus.cc:2468] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8a2e7dba69c4c80bb3672eeaf63e1c2 in term 1.
I20251211 08:58:42.411338 17229 leader_election.cc:304] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2; no voters:
I20251211 08:58:42.414003 17794 raft_consensus.cc:515] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.417325 17296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ee60b7cb9094553af8be206d4e38c22" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2"
I20251211 08:58:42.417835 17296 raft_consensus.cc:3060] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Advancing to term 1
W20251211 08:58:42.419063 17230 leader_election.cc:343] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): Not found: Tablet not found: 1ca4a92a351b4a4fa77ef805bfcdb451
I20251211 08:58:42.419902 17742 leader_election.cc:290] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [CANDIDATE]: Term 1 election: Requested vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:42.420432 17742 raft_consensus.cc:2804] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:42.420838 17742 raft_consensus.cc:697] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 LEADER]: Becoming Leader. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Running, Role: LEADER
I20251211 08:58:42.421589 17742 consensus_queue.cc:237] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:42.433311 17296 raft_consensus.cc:2468] T 0ee60b7cb9094553af8be206d4e38c22 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62329f65aae54e84b396a51dd81f1b70 in term 1.
I20251211 08:58:42.435851 17449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ee60b7cb9094553af8be206d4e38c22" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf"
I20251211 08:58:42.434481 17794 leader_election.cc:290] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 election: Requested vote from peers e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:42.436375 17449 raft_consensus.cc:3060] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:42.434478 17070 leader_election.cc:304] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2; no voters:
I20251211 08:58:42.438038 17794 raft_consensus.cc:2804] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:42.442929 17449 raft_consensus.cc:2468] T 0ee60b7cb9094553af8be206d4e38c22 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62329f65aae54e84b396a51dd81f1b70 in term 1.
I20251211 08:58:42.443503 17794 raft_consensus.cc:697] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [term 1 LEADER]: Becoming Leader. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Running, Role: LEADER
I20251211 08:58:42.444329 17794 consensus_queue.cc:237] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
W20251211 08:58:42.515894 17382 connection.cc:537] server connection from 127.0.0.1:52100 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:42.516009 17071 connection.cc:537] server connection from 127.0.0.1:43306 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:42.516659 17232 connection.cc:537] server connection from 127.0.0.1:37282 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:42.516723 17504 connection.cc:537] client connection to 127.13.146.62:43161 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:42.518301 17504 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.13.146.62:43161)
W20251211 08:58:42.519331 17504 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.13.146.62:43161)
W20251211 08:58:42.520351 17504 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.13.146.62:43161)
W20251211 08:58:42.520927 17504 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.146.62:43161)
I20251211 08:58:42.657513 17816 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 133269af8ce44b579ebbb7805bc58800)
I20251211 08:58:42.657922 17816 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:42.660478 17816 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
W20251211 08:58:42.664256 17678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:58:42.668452 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:58:42.669023 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035): Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:42.669358 17678 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8dd1b89d5ad4e529459606ce6f85ab8; no voters: 133269af8ce44b579ebbb7805bc58800, c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:42.669983 17816 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251211 08:58:42.745832 17820 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:42.746222 17820 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:42.747557 17820 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 08:58:42.749060 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 08:58:42.749562 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:58:42.749815 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 08:58:42.750340 17820 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:43.223218 17821 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 133269af8ce44b579ebbb7805bc58800)
I20251211 08:58:43.223735 17821 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:43.225735 17821 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
W20251211 08:58:43.233013 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:58:43.234532 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035): Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:43.235278 17678 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8dd1b89d5ad4e529459606ce6f85ab8; no voters: 133269af8ce44b579ebbb7805bc58800, c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:43.236542 17821 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251211 08:58:43.599294 17825 consensus_queue.cc:1048] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251211 08:58:43.616315 17230 consensus_peers.cc:597] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2 -> Peer 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): Couldn't send request to peer 7467cf4890bc486bbfb9b8f4bea2c2cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ca4a92a351b4a4fa77ef805bfcdb451. This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:43.642972 17828 consensus_queue.cc:1048] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:43.666504 17838 consensus_queue.cc:1048] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:44.200050 17847 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 133269af8ce44b579ebbb7805bc58800)
I20251211 08:58:44.200403 17847 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:44.201797 17847 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
W20251211 08:58:44.205806 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:58:44.206487 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035): Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:44.206810 17678 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8dd1b89d5ad4e529459606ce6f85ab8; no voters: 133269af8ce44b579ebbb7805bc58800, c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:44.207316 17847 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251211 08:58:44.414621 17852 ts_tablet_manager.cc:933] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Initiating tablet copy from peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
I20251211 08:58:44.415946 17852 tablet_copy_client.cc:323] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet copy: Beginning tablet copy session from remote peer at address 127.13.146.2:40951
I20251211 08:58:44.417677 17316 tablet_copy_service.cc:140] P e8a2e7dba69c4c80bb3672eeaf63e1c2: Received BeginTabletCopySession request for tablet 1ca4a92a351b4a4fa77ef805bfcdb451 from peer 7467cf4890bc486bbfb9b8f4bea2c2cf ({username='slave'} at 127.13.146.3:35069)
I20251211 08:58:44.418109 17316 tablet_copy_service.cc:161] P e8a2e7dba69c4c80bb3672eeaf63e1c2: Beginning new tablet copy session on tablet 1ca4a92a351b4a4fa77ef805bfcdb451 from peer 7467cf4890bc486bbfb9b8f4bea2c2cf at {username='slave'} at 127.13.146.3:35069: session id = 7467cf4890bc486bbfb9b8f4bea2c2cf-1ca4a92a351b4a4fa77ef805bfcdb451
I20251211 08:58:44.423178 17316 tablet_copy_source_session.cc:215] T 1ca4a92a351b4a4fa77ef805bfcdb451 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Tablet Copy: opened 0 blocks and 1 log segments
I20251211 08:58:44.425554 17852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca4a92a351b4a4fa77ef805bfcdb451. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:44.434476 17852 tablet_copy_client.cc:806] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet copy: Starting download of 0 data blocks...
I20251211 08:58:44.434906 17852 tablet_copy_client.cc:670] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet copy: Starting download of 1 WAL segments...
I20251211 08:58:44.437798 17852 tablet_copy_client.cc:538] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251211 08:58:44.443105 17852 tablet_bootstrap.cc:492] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
I20251211 08:58:44.454576 17852 tablet_bootstrap.cc:492] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251211 08:58:44.455295 17852 tablet_bootstrap.cc:492] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap complete.
I20251211 08:58:44.455791 17852 ts_tablet_manager.cc:1403] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20251211 08:58:44.457525 17852 raft_consensus.cc:359] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:44.457973 17852 raft_consensus.cc:740] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:44.458410 17852 consensus_queue.cc:260] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:44.460767 17852 ts_tablet_manager.cc:1434] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251211 08:58:44.463879 17316 tablet_copy_service.cc:342] P e8a2e7dba69c4c80bb3672eeaf63e1c2: Request end of tablet copy session 7467cf4890bc486bbfb9b8f4bea2c2cf-1ca4a92a351b4a4fa77ef805bfcdb451 received from {username='slave'} at 127.13.146.3:35069
I20251211 08:58:44.464344 17316 tablet_copy_service.cc:434] P e8a2e7dba69c4c80bb3672eeaf63e1c2: ending tablet copy session 7467cf4890bc486bbfb9b8f4bea2c2cf-1ca4a92a351b4a4fa77ef805bfcdb451 on tablet 1ca4a92a351b4a4fa77ef805bfcdb451 with peer 7467cf4890bc486bbfb9b8f4bea2c2cf
I20251211 08:58:44.579540 13896 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
/tmp/dist-test-taskt39ak2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.61:36035
--webserver_interface=127.13.146.61
--webserver_port=41841
--builtin_ntp_servers=127.13.146.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--hive_metastore_uris=thrift://127.0.0.1:46123
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251211 08:58:44.887809 17857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251211 08:58:44.888368 17857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251211 08:58:44.888788 17857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251211 08:58:44.920053 17857 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20251211 08:58:44.920339 17857 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20251211 08:58:44.920581 17857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251211 08:58:44.920805 17857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251211 08:58:44.921038 17857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251211 08:58:44.921257 17857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251211 08:58:44.956050 17857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.146.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:46123
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.146.61:36035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.146.61
--webserver_port=41841
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 11 Dec 2025 08:43:21 UTC on 5fd53c4cbb9d
build id 9461
TSAN enabled
I20251211 08:58:44.957398 17857 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251211 08:58:44.958923 17857 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251211 08:58:44.969148 17870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:45.385751 17449 raft_consensus.cc:1217] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20251211 08:58:45.825281 17887 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 133269af8ce44b579ebbb7805bc58800)
I20251211 08:58:45.826107 17887 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:45.834942 17887 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
W20251211 08:58:45.862723 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 08:58:45.863025 17882 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251211 08:58:45.863793 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035): Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:45.863956 17882 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:45.864368 17678 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8dd1b89d5ad4e529459606ce6f85ab8; no voters: 133269af8ce44b579ebbb7805bc58800, c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:45.865469 17887 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251211 08:58:45.870231 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 08:58:45.874353 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:58:45.871470 17882 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
I20251211 08:58:45.874861 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 08:58:45.876065 17882 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:46.581295 17554 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:58:46.581784 17554 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:58:46.636484 17899 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 133269af8ce44b579ebbb7805bc58800)
I20251211 08:58:46.637730 17899 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
W20251211 08:58:46.660910 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 08:58:46.667636 17899 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
W20251211 08:58:46.669811 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035): Network error: Client connection negotiation failed: client connection to 127.13.146.61:36035: connect: Connection refused (error 111)
I20251211 08:58:46.670420 17678 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8dd1b89d5ad4e529459606ce6f85ab8; no voters: 133269af8ce44b579ebbb7805bc58800, c3112fc6f4ae4541a2af2405a97c707b
I20251211 08:58:46.671839 17899 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251211 08:58:46.372908 17869 debug-util.cc:398] Leaking SignalData structure 0x7b0800028120 after lost signal to thread 17857
W20251211 08:58:46.932823 17857 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.959s user 0.697s sys 1.159s
W20251211 08:58:46.933182 17857 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.960s user 0.697s sys 1.159s
W20251211 08:58:44.973191 17871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251211 08:58:46.935398 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251211 08:58:46.939653 17857 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20251211 08:58:46.939572 17874 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1966 milliseconds
I20251211 08:58:46.941177 17857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251211 08:58:46.944228 17857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251211 08:58:46.945660 17857 hybrid_clock.cc:648] HybridClock initialized: now 1765443526945587 us; error 64 us; skew 500 ppm
I20251211 08:58:46.946496 17857 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251211 08:58:46.951975 17857 webserver.cc:492] Webserver started at http://127.13.146.61:41841/ using document root <none> and password file <none>
I20251211 08:58:46.952739 17857 fs_manager.cc:362] Metadata directory not provided
I20251211 08:58:46.952908 17857 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251211 08:58:46.960968 17857 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20251211 08:58:46.965919 17910 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251211 08:58:46.966991 17857 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251211 08:58:46.967283 17857 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data,/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal
uuid: "c3112fc6f4ae4541a2af2405a97c707b"
format_stamp: "Formatted at 2025-12-11 08:58:26 on dist-test-slave-0dfs"
I20251211 08:58:46.969434 17857 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251211 08:58:47.028343 17857 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251211 08:58:47.029844 17857 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251211 08:58:47.030265 17857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251211 08:58:47.113379 17857 rpc_server.cc:307] RPC server started. Bound to: 127.13.146.61:36035
I20251211 08:58:47.113461 17961 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.146.61:36035 every 8 connection(s)
I20251211 08:58:47.116257 17857 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/master-1/data/info.pb
I20251211 08:58:47.124480 13896 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskt39ak2/build/tsan/bin/kudu as pid 17857
I20251211 08:58:47.127377 17962 sys_catalog.cc:263] Verifying existing consensus state
I20251211 08:58:47.134366 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Bootstrap starting.
I20251211 08:58:47.159974 17962 log.cc:826] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Log is configured to *not* fsync() on all Append() calls
I20251211 08:58:47.222175 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=15 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251211 08:58:47.223007 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: Bootstrap complete.
I20251211 08:58:47.239663 17962 raft_consensus.cc:359] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:47.241592 17962 raft_consensus.cc:740] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3112fc6f4ae4541a2af2405a97c707b, State: Initialized, Role: FOLLOWER
I20251211 08:58:47.242388 17962 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 6.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:47.244707 17967 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:47.245478 17967 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:47.245802 17962 sys_catalog.cc:565] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: configured and running, proceeding with master startup.
I20251211 08:58:47.297088 17981 catalog_manager.cc:1269] Loaded cluster ID: c258c1a306e24c4197d31ee43870f1fb
I20251211 08:58:47.297456 17981 catalog_manager.cc:1562] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: loading cluster ID for follower catalog manager: success
I20251211 08:58:47.279031 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:58:47.304288 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:58:47.307911 17981 catalog_manager.cc:1584] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: acquiring CA information for follower catalog manager: success
I20251211 08:58:47.314507 17981 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251211 08:58:47.406509 17184 heartbeater.cc:344] Connected to a master server at 127.13.146.61:36035
I20251211 08:58:47.416144 17927 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" instance_seqno: 1765443513006607) as {username='slave'} at 127.13.146.1:38985; Asking this server to re-register.
I20251211 08:58:47.417829 17184 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:47.421625 17927 ts_manager.cc:194] Registered new tserver with Master: 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231)
I20251211 08:58:47.471547 17495 heartbeater.cc:344] Connected to a master server at 127.13.146.61:36035
I20251211 08:58:47.475629 17927 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" instance_seqno: 1765443517190554) as {username='slave'} at 127.13.146.3:35185; Asking this server to re-register.
I20251211 08:58:47.476908 17495 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:47.480777 17927 ts_manager.cc:194] Registered new tserver with Master: 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:47.490588 17342 heartbeater.cc:344] Connected to a master server at 127.13.146.61:36035
I20251211 08:58:47.494908 17927 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" instance_seqno: 1765443515305506) as {username='slave'} at 127.13.146.2:42893; Asking this server to re-register.
I20251211 08:58:47.496106 17342 heartbeater.cc:461] Registering TS with master...
I20251211 08:58:47.499226 17927 ts_manager.cc:194] Registered new tserver with Master: e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
I20251211 08:58:47.694727 17967 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:47.695338 17967 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
W20251211 08:58:47.701840 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 08:58:47.701965 17967 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:58:47.706661 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 08:58:47.711753 17703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 7 candidate_status { last_received { term: 6 index: 27 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:47.712349 17703 raft_consensus.cc:2393] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3112fc6f4ae4541a2af2405a97c707b in current term 7: Already voted for candidate 133269af8ce44b579ebbb7805bc58800 in this term.
I20251211 08:58:47.713399 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:47.714102 17967 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 6 FOLLOWER]: Advancing to term 7
I20251211 08:58:47.723224 17967 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251211 08:58:48.285729 17997 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:48.286111 17997 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:48.287631 17997 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:48.289382 17703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 8 candidate_status { last_received { term: 6 index: 27 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:48.290273 17703 raft_consensus.cc:2410] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3112fc6f4ae4541a2af2405a97c707b for term 8 because replica has last-logged OpId of term: 7 index: 36, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 27.
W20251211 08:58:48.293548 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 08:58:48.293879 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:48.294392 17997 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251211 08:58:49.113965 18006 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:49.114300 18006 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:49.116591 18006 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
I20251211 08:58:49.116780 17703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3112fc6f4ae4541a2af2405a97c707b" candidate_term: 8 candidate_status { last_received { term: 6 index: 27 } } ignore_live_leader: false dest_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" is_pre_election: true
I20251211 08:58:49.117580 17703 raft_consensus.cc:2410] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3112fc6f4ae4541a2af2405a97c707b for term 8 because replica has last-logged OpId of term: 7 index: 36, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 27.
W20251211 08:58:49.121373 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 08:58:49.121637 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:49.122156 18006 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251211 08:58:49.777488 18013 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 133269af8ce44b579ebbb7805bc58800)
I20251211 08:58:49.778132 18013 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:49.777515 13896 master-stress-test.cc:577] Killing test thread 1/10
I20251211 08:58:49.780108 18013 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
W20251211 08:58:49.784567 17678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251211 08:58:49.787743 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 08:58:49.792199 17937 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 8 candidate_status { last_received { term: 7 index: 36 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b" is_pre_election: true
I20251211 08:58:49.792775 17937 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8dd1b89d5ad4e529459606ce6f85ab8 in term 7.
I20251211 08:58:49.793638 17678 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800
I20251211 08:58:49.794214 18013 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251211 08:58:49.794514 18013 raft_consensus.cc:493] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 133269af8ce44b579ebbb7805bc58800)
I20251211 08:58:49.794783 18013 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 7 FOLLOWER]: Advancing to term 8
I20251211 08:58:49.800237 18013 raft_consensus.cc:515] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:49.801609 18013 leader_election.cc:290] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 election: Requested vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), c3112fc6f4ae4541a2af2405a97c707b (127.13.146.61:36035)
I20251211 08:58:49.802727 17937 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" candidate_term: 8 candidate_status { last_received { term: 7 index: 36 } } ignore_live_leader: false dest_uuid: "c3112fc6f4ae4541a2af2405a97c707b"
I20251211 08:58:49.803174 17937 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 7 FOLLOWER]: Advancing to term 8
W20251211 08:58:49.804418 17678 leader_election.cc:336] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 08:58:49.807852 17937 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8dd1b89d5ad4e529459606ce6f85ab8 in term 8.
I20251211 08:58:49.808576 17678 leader_election.cc:304] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b8dd1b89d5ad4e529459606ce6f85ab8, c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800
I20251211 08:58:49.809278 18013 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 8 FOLLOWER]: Leader election won for term 8
I20251211 08:58:49.809301 18018 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:49.809715 18018 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:49.810413 18013 raft_consensus.cc:697] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [term 8 LEADER]: Becoming Leader. State: Replica: b8dd1b89d5ad4e529459606ce6f85ab8, State: Running, Role: LEADER
I20251211 08:58:49.811268 18018 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
I20251211 08:58:49.811307 18013 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 7.36, Last appended by leader: 36, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
W20251211 08:58:49.812608 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 08:58:49.813234 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:58:49.813588 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 08:58:49.814303 18018 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:49.816258 18019 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8dd1b89d5ad4e529459606ce6f85ab8. Latest consensus state: current_term: 8 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:49.816809 18019 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: LEADER
I20251211 08:58:50.309146 17937 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Refusing update from remote peer b8dd1b89d5ad4e529459606ce6f85ab8: Log matching property violated. Preceding OpId in replica: term: 6 index: 27. Preceding OpId from leader: term: 8 index: 37. (index mismatch)
I20251211 08:58:50.310427 18019 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 27, Time since last communication: 0.000s
I20251211 08:58:50.347160 18027 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8dd1b89d5ad4e529459606ce6f85ab8. Latest consensus state: current_term: 8 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:50.352109 18027 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:50.350939 18028 mvcc.cc:204] Tried to move back new op lower bound from 7231256666565349376 to 7231256664308768768. Current Snapshot: MvccSnapshot[applied={T|T < 7231256666402648064}]
I20251211 08:58:50.359056 18031 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:50.359918 18031 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:50.360376 18027 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:50.360996 18027 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [sys.catalog]: This master's current role is: FOLLOWER
I20251211 08:58:50.366025 18019 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } } }
I20251211 08:58:50.366684 18019 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 [sys.catalog]: This master's current role is: LEADER
W20251211 08:58:50.372141 17678 consensus_peers.cc:597] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8 -> Peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Couldn't send request to peer 133269af8ce44b579ebbb7805bc58800. Status: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251211 08:58:50.464350 18021 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251211 08:58:50.472759 18021 catalog_manager.cc:679] Loaded metadata for table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]
I20251211 08:58:50.474270 18021 catalog_manager.cc:679] Loaded metadata for table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4]
I20251211 08:58:50.475953 18021 catalog_manager.cc:679] Loaded metadata for table default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283]
I20251211 08:58:50.481096 18021 tablet_loader.cc:96] loaded metadata for tablet 0ee60b7cb9094553af8be206d4e38c22 (table default.table_12ea97bf52084b09955460036752d6e4 [id=9d454284fb154c6297fc7ea77ed02283])
I20251211 08:58:50.482088 18021 tablet_loader.cc:96] loaded metadata for tablet 1ca4a92a351b4a4fa77ef805bfcdb451 (table default.table_d36118f54fec423badd3d2b59d64b05c [id=53d392cd36244b498c2740a372a1daf4])
I20251211 08:58:50.483417 18021 tablet_loader.cc:96] loaded metadata for tablet 316fbbd2f6e0476e9b9768b954a21fbc (table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299])
I20251211 08:58:50.484843 18021 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251211 08:58:50.488356 18021 catalog_manager.cc:1269] Loaded cluster ID: c258c1a306e24c4197d31ee43870f1fb
I20251211 08:58:50.488533 18021 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251211 08:58:50.492094 18021 catalog_manager.cc:1514] Loading token signing keys...
I20251211 08:58:50.495219 18021 catalog_manager.cc:6038] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Loaded TSK: 0
I20251211 08:58:50.496130 18021 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251211 08:58:50.499325 18021 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251211 08:58:50.502969 18021 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251211 08:58:50.613423 17694 master_service.cc:954] ReplaceTablet: received request to replace tablet 316fbbd2f6e0476e9b9768b954a21fbc from {username='slave'} at 127.0.0.1:33992
I20251211 08:58:50.613749 17693 master_service.cc:954] ReplaceTablet: received request to replace tablet 316fbbd2f6e0476e9b9768b954a21fbc from {username='slave'} at 127.0.0.1:33992
I20251211 08:58:50.614924 17692 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33992:
table { table_name: "default.Table_da60da9af7664a10a99d2fa227af8235" } new_table_name: "default.Table_59e9546b87cc408f961a8f029510dc53" modify_external_catalogs: true
I20251211 08:58:50.615518 17690 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33992:
table { table_name: "default.Table_12ea97bf52084b09955460036752d6e4" } modify_external_catalogs: true
I20251211 08:58:50.616513 17690 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33992:
table { table_name: "default.Table_12ea97bf52084b09955460036752d6e4" } modify_external_catalogs: true
I20251211 08:58:50.615681 17691 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33992:
table { table_name: "default.Table_d36118f54fec423badd3d2b59d64b05c" } new_table_name: "default.Table_c00e0c8e65c24aac8676a17546efe6eb" modify_external_catalogs: true
I20251211 08:58:50.616982 17689 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "default.Table_eff061b04cb14ce88f2a8555db7504ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:50.619542 17688 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "default.Table_097cc85012564d15aa8db749266d3068"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:50.619237 17687 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "default.Table_9088340c00c94ec8bb9312a333dec04d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251211 08:58:50.620931 17686 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "default.Table_7bc946992a8e44a0987ff88263b4c255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251211 08:58:50.626053 17753 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1765443517, eventType=CREATE_TABLE, dbName=default, tableName=table_1bf93acd2af64f98b4cda7ce375e702e, message={"server":"","servicePrincipal":"","db":"default","table":"table_1bf93acd2af64f98b4cda7ce375e702e","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_1bf93acd2af64f98b4cda7ce375e702e\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765443517},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/warehouse/table_1bf93acd2af64f98b4cda7ce375e702e\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_1bf93acd2af64f98b4cda7ce375e702e\",\"kudu.table_id\":\"86b322f346e0458daf6c4f9dd962de20\",\"transient_lastDdlTime\":\"1765443517\",\"comment\":\"\",\"kudu.master_addresses\":\"127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"c258c1a306e24c4197d31ee43870f1fb\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765443517,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765443517, eventType=CREATE_TABLE, dbName=default, tableName=table_d36118f54fec423badd3d2b59d64b05c, message={"server":"","servicePrincipal":"","db":"default","table":"table_d36118f54fec423badd3d2b59d64b05c","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d36118f54fec423badd3d2b59d64b05c\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765443517},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/warehouse/table_d36118f54fec423badd3d2b59d64b05c\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d36118f54fec423badd3d2b59d64b05c\",\"kudu.table_id\":\"53d392cd36244b498c2740a372a1daf4\",\"transient_lastDdlTime\":\"1765443517\",\"comment\":\"\",\"kudu.master_addresses\":\"127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"c258c1a306e24c4197d31ee43870f1fb\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765443517,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251211 08:58:50.633421 17694 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b8dd1b89d5ad4e529459606ce6f85ab8: Sending DeleteTablet for 3 replicas of tablet 316fbbd2f6e0476e9b9768b954a21fbc
I20251211 08:58:50.637490 17694 catalog_manager.cc:6724] ReplaceTablet: tablet 316fbbd2f6e0476e9b9768b954a21fbc deleted and replaced by tablet 1ba479775bae4cb09c5267315f7acf66
I20251211 08:58:50.651479 17117 tablet_service.cc:1552] Processing DeleteTablet for tablet 316fbbd2f6e0476e9b9768b954a21fbc with delete_type TABLET_DATA_DELETED (replaced by tablet 1ba479775bae4cb09c5267315f7acf66) from {username='slave'} at 127.0.0.1:59938
I20251211 08:58:50.655722 17276 tablet_service.cc:1552] Processing DeleteTablet for tablet 316fbbd2f6e0476e9b9768b954a21fbc with delete_type TABLET_DATA_DELETED (replaced by tablet 1ba479775bae4cb09c5267315f7acf66) from {username='slave'} at 127.0.0.1:39994
I20251211 08:58:50.657531 17428 tablet_service.cc:1552] Processing DeleteTablet for tablet 316fbbd2f6e0476e9b9768b954a21fbc with delete_type TABLET_DATA_DELETED (replaced by tablet 1ba479775bae4cb09c5267315f7acf66) from {username='slave'} at 127.0.0.1:47202
I20251211 08:58:50.659312 18049 tablet_replica.cc:333] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf: stopping tablet replica
I20251211 08:58:50.660279 18049 raft_consensus.cc:2243] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:50.660892 18049 raft_consensus.cc:2272] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:50.663316 18049 ts_tablet_manager.cc:1916] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:50.667969 18048 tablet_replica.cc:333] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2: stopping tablet replica
I20251211 08:58:50.668840 18048 raft_consensus.cc:2243] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 LEADER]: Raft consensus shutting down.
I20251211 08:58:50.669899 18048 raft_consensus.cc:2272] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:50.671926 18048 ts_tablet_manager.cc:1916] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:50.672410 18047 tablet_replica.cc:333] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70: stopping tablet replica
I20251211 08:58:50.673133 18047 raft_consensus.cc:2243] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251211 08:58:50.673434 17276 tablet_service.cc:1505] Processing CreateTablet for tablet 1ba479775bae4cb09c5267315f7acf66 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:50.674593 17276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba479775bae4cb09c5267315f7acf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:50.675433 17118 tablet_service.cc:1505] Processing CreateTablet for tablet 1ba479775bae4cb09c5267315f7acf66 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:50.677002 17118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba479775bae4cb09c5267315f7acf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:50.677160 17428 tablet_service.cc:1505] Processing CreateTablet for tablet 1ba479775bae4cb09c5267315f7acf66 (DEFAULT_TABLE table=default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251211 08:58:50.678463 17428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba479775bae4cb09c5267315f7acf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251211 08:58:50.680840 18049 ts_tablet_manager.cc:1929] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:50.681282 18049 log.cc:1199] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-2/wal/wals/316fbbd2f6e0476e9b9768b954a21fbc
I20251211 08:58:50.682159 18049 ts_tablet_manager.cc:1950] T 316fbbd2f6e0476e9b9768b954a21fbc P 7467cf4890bc486bbfb9b8f4bea2c2cf: Deleting consensus metadata
I20251211 08:58:50.673770 18047 raft_consensus.cc:2272] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251211 08:58:50.689549 18047 ts_tablet_manager.cc:1916] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251211 08:58:50.696700 17679 catalog_manager.cc:4985] TS 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219): tablet 316fbbd2f6e0476e9b9768b954a21fbc (table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]) successfully deleted
I20251211 08:58:50.699909 18051 tablet_bootstrap.cc:492] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70: Bootstrap starting.
I20251211 08:58:50.704836 18050 tablet_bootstrap.cc:492] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Bootstrap starting.
I20251211 08:58:50.706112 18052 tablet_bootstrap.cc:492] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Bootstrap starting.
I20251211 08:58:50.706298 18051 tablet_bootstrap.cc:654] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:50.712234 18052 tablet_bootstrap.cc:654] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:50.714624 18048 ts_tablet_manager.cc:1929] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:50.716707 18047 ts_tablet_manager.cc:1929] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251211 08:58:50.717159 18047 log.cc:1199] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-0/wal/wals/316fbbd2f6e0476e9b9768b954a21fbc
I20251211 08:58:50.717288 18048 log.cc:1199] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting WAL directory at /tmp/dist-test-taskt39ak2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1765443426576974-13896-0/minicluster-data/ts-1/wal/wals/316fbbd2f6e0476e9b9768b954a21fbc
I20251211 08:58:50.718055 18047 ts_tablet_manager.cc:1950] T 316fbbd2f6e0476e9b9768b954a21fbc P 62329f65aae54e84b396a51dd81f1b70: Deleting consensus metadata
I20251211 08:58:50.718398 18051 tablet_bootstrap.cc:492] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70: No bootstrap required, opened a new log
I20251211 08:58:50.718844 18051 ts_tablet_manager.cc:1403] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.012s
I20251211 08:58:50.719455 18048 ts_tablet_manager.cc:1950] T 316fbbd2f6e0476e9b9768b954a21fbc P e8a2e7dba69c4c80bb3672eeaf63e1c2: Deleting consensus metadata
I20251211 08:58:50.721773 17678 catalog_manager.cc:4985] TS 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): tablet 316fbbd2f6e0476e9b9768b954a21fbc (table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]) successfully deleted
I20251211 08:58:50.721884 18051 raft_consensus.cc:359] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:50.722611 18051 raft_consensus.cc:385] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
@ 0x7f4b8fb25950 google::LogMessage::Flush() at ??:0
I20251211 08:58:50.722997 18051 raft_consensus.cc:740] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Initialized, Role: FOLLOWER
I20251211 08:58:50.724164 18052 tablet_bootstrap.cc:492] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2: No bootstrap required, opened a new log
I20251211 08:58:50.723742 18051 consensus_queue.cc:260] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:50.724570 18052 ts_tablet_manager.cc:1403] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20251211 08:58:50.727526 17678 catalog_manager.cc:4985] TS e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): tablet 316fbbd2f6e0476e9b9768b954a21fbc (table default.table_da60da9af7664a10a99d2fa227af8235 [id=0869b45d01994929ba2c3284c458c299]) successfully deleted
I20251211 08:58:50.727348 18052 raft_consensus.cc:359] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:50.728008 18052 raft_consensus.cc:385] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:50.728253 18052 raft_consensus.cc:740] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a2e7dba69c4c80bb3672eeaf63e1c2, State: Initialized, Role: FOLLOWER
I20251211 08:58:50.728652 18052 consensus_queue.cc:260] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:50.737926 18050 tablet_bootstrap.cc:654] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Neither blocks nor log segments found. Creating new log.
I20251211 08:58:50.738057 18052 ts_tablet_manager.cc:1434] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20251211 08:58:50.738713 17341 heartbeater.cc:499] Master 127.13.146.60:34701 was elected leader, sending a full tablet report...
I20251211 08:58:50.739037 18051 ts_tablet_manager.cc:1434] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20251211 08:58:50.739780 17183 heartbeater.cc:499] Master 127.13.146.60:34701 was elected leader, sending a full tablet report...
@ 0x7f4b8fb256bf google::LogMessage::~LogMessage() at ??:0
I20251211 08:58:50.752643 18050 tablet_bootstrap.cc:492] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf: No bootstrap required, opened a new log
I20251211 08:58:50.753041 18050 ts_tablet_manager.cc:1403] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.007s
I20251211 08:58:50.754190 17694 catalog_manager.cc:5654] T 0ee60b7cb9094553af8be206d4e38c22 P 62329f65aae54e84b396a51dd81f1b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1). New cstate: current_term: 1 leader_uuid: "62329f65aae54e84b396a51dd81f1b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } health_report { overall_health: HEALTHY } } }
I20251211 08:58:50.755968 18050 raft_consensus.cc:359] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:50.756798 18050 raft_consensus.cc:385] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251211 08:58:50.757090 18050 raft_consensus.cc:740] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467cf4890bc486bbfb9b8f4bea2c2cf, State: Initialized, Role: FOLLOWER
I20251211 08:58:50.756438 17694 catalog_manager.cc:5654] T 1ca4a92a351b4a4fa77ef805bfcdb451 P 62329f65aae54e84b396a51dd81f1b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2). New cstate: current_term: 1 leader_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } }
I20251211 08:58:50.757741 18050 consensus_queue.cc:260] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
@ 0x7f4b9abb826c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251211 08:58:50.767709 18050 ts_tablet_manager.cc:1434] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf: Time spent starting tablet: real 0.014s user 0.003s sys 0.009s
I20251211 08:58:50.778909 17494 heartbeater.cc:499] Master 127.13.146.60:34701 was elected leader, sending a full tablet report...
W20251211 08:58:50.783172 17694 catalog_manager.cc:5487] Ignoring report from unknown tablet 128b128c966b4dcfa6241b96e3ae61b1
@ 0x7f4b9abb7a2c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f4b9abbbaa2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f4b9abbba5a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f4b9abbb9ea std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f4b9abbb9b2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f4b9abbacae std::__1::__function::__func<>::operator()() at ??:0
@ 0x5aa2b5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5aa0ea std::__1::function<>::operator()() at ??:?
@ 0x7f4b91d473ab kudu::Thread::SuperviseThread() at ??:0
@ 0x474190 __tsan_thread_start_func at ??:?
@ 0x7f4b934526db start_thread at ??:0
@ 0x7f4b8e7ca71f clone at ??:0
I20251211 08:58:50.947938 18026 raft_consensus.cc:493] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:50.948307 18026 raft_consensus.cc:515] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:50.950083 18026 leader_election.cc:290] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:50.950968 17296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ba479775bae4cb09c5267315f7acf66" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" is_pre_election: true
I20251211 08:58:50.951203 17449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ba479775bae4cb09c5267315f7acf66" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" is_pre_election: true
I20251211 08:58:50.951450 17296 raft_consensus.cc:2468] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62329f65aae54e84b396a51dd81f1b70 in term 0.
I20251211 08:58:50.951851 17449 raft_consensus.cc:2468] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62329f65aae54e84b396a51dd81f1b70 in term 0.
I20251211 08:58:50.952333 17070 leader_election.cc:304] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2; no voters:
I20251211 08:58:50.952997 18026 raft_consensus.cc:2804] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251211 08:58:50.953281 18026 raft_consensus.cc:493] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251211 08:58:50.953533 18026 raft_consensus.cc:3060] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:50.958123 18026 raft_consensus.cc:515] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
I20251211 08:58:50.959863 18026 leader_election.cc:290] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 election: Requested vote from peers e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951), 7467cf4890bc486bbfb9b8f4bea2c2cf (127.13.146.3:41219)
I20251211 08:58:50.960731 17449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ba479775bae4cb09c5267315f7acf66" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf"
I20251211 08:58:50.960623 17296 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ba479775bae4cb09c5267315f7acf66" candidate_uuid: "62329f65aae54e84b396a51dd81f1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2"
I20251211 08:58:50.961089 17449 raft_consensus.cc:3060] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:50.961109 17296 raft_consensus.cc:3060] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 0 FOLLOWER]: Advancing to term 1
I20251211 08:58:50.966445 17449 raft_consensus.cc:2468] T 1ba479775bae4cb09c5267315f7acf66 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62329f65aae54e84b396a51dd81f1b70 in term 1.
I20251211 08:58:50.967337 17071 leader_election.cc:304] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62329f65aae54e84b396a51dd81f1b70, 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters:
I20251211 08:58:50.968070 18026 raft_consensus.cc:2804] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [term 1 FOLLOWER]: Leader election won for term 1
I20251211 08:58:50.968225 17296 raft_consensus.cc:2468] T 1ba479775bae4cb09c5267315f7acf66 P e8a2e7dba69c4c80bb3672eeaf63e1c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62329f65aae54e84b396a51dd81f1b70 in term 1.
I20251211 08:58:50.969928 18026 raft_consensus.cc:697] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [term 1 LEADER]: Becoming Leader. State: Replica: 62329f65aae54e84b396a51dd81f1b70, State: Running, Role: LEADER
I20251211 08:58:50.971171 18026 consensus_queue.cc:237] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } }
W20251211 08:58:51.087962 17072 connection.cc:537] client connection to 127.13.146.60:34701 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251211 08:58:51.091148 17506 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.146.60:34701)
W20251211 08:58:51.093030 17506 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.13.146.60:34701)
W20251211 08:58:51.093572 17506 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.13.146.60:34701)
I20251211 08:58:51.277758 18074 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:51.278236 18074 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:51.279874 18074 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:58:51.284334 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:58:51.284986 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:51.285305 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:51.286155 18074 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 08:58:51.800577 18026 consensus_queue.cc:1048] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:51.816583 18082 consensus_queue.cc:1048] T 1ba479775bae4cb09c5267315f7acf66 P 62329f65aae54e84b396a51dd81f1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251211 08:58:52.022334 18091 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:52.022670 18091 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:52.024353 18091 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:58:52.028287 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:58:52.028733 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:52.029114 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:52.029634 18091 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 08:58:52.306407 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:58:52.306747 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:58:52.718456 18104 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:52.718850 18104 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:52.720516 18104 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:58:52.721779 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20251211 08:58:52.724134 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:58:52.724575 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:52.724861 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:52.725310 18104 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 08:58:53.503602 18119 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:53.503968 18119 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:53.505357 18119 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 08:58:53.506748 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 08:58:53.507315 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:58:53.507599 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 08:58:53.508026 18119 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:58:53.603106 18120 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:53.603541 18120 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:53.605170 18120 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:58:53.610002 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:58:53.610431 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:53.610731 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:53.611282 18120 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 08:58:55.750523 18147 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:55.751186 18147 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:55.755330 18147 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:58:55.758955 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:58:55.760046 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:58:55.760531 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:55.762066 18147 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 08:58:57.309361 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:58:57.309748 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:58:58.095994 18181 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:58:58.096413 18181 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:58:58.097707 18181 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:58:58.099507 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20251211 08:58:58.102504 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 08:58:58.103013 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 08:58:58.103338 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:58:58.103783 18181 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 08:58:58.724629 18192 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:58:58.725167 18192 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:58:58.727265 18192 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 08:58:58.728585 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 08:58:58.729061 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:58:58.729301 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 08:58:58.729784 18192 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:59:00.581655 18206 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:59:00.582028 18206 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:59:00.583391 18206 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:59:00.587782 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:59:00.588285 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:59:00.588559 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:59:00.589031 18206 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 08:59:02.312294 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:02.312619 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:59:07.315785 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:07.316188 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:59:08.655759 18309 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:59:08.656213 18309 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:59:08.657629 18309 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:59:08.660055 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251211 08:59:08.662873 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 08:59:08.663229 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 08:59:08.663508 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:59:08.664031 18309 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 08:59:10.703258 18339 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:59:10.703773 18339 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:59:10.705163 18339 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 08:59:10.706769 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 08:59:10.707234 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:59:10.707453 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 08:59:10.707943 18339 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:59:12.318737 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:12.319132 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:59:14.645853 18395 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:59:14.646198 18395 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:59:14.647964 18395 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 08:59:14.649261 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 08:59:14.649667 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:59:14.649873 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 08:59:14.650269 18395 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:59:15.108585 18403 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:59:15.109130 18403 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:59:15.110616 18403 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:59:15.112725 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 08:59:15.117957 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:59:15.118520 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:59:15.118857 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:59:15.119441 18403 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251211 08:59:15.360738 17958 debug-util.cc:398] Leaking SignalData structure 0x7b080006f1e0 after lost signal to thread 17865
W20251211 08:59:15.361435 17958 debug-util.cc:398] Leaking SignalData structure 0x7b080006f760 after lost signal to thread 17961
I20251211 08:59:17.360369 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:17.360849 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:59:22.363445 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:22.363874 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:59:23.030303 18506 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:59:23.030638 18506 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:59:23.031982 18506 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:59:23.033953 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 08:59:23.037897 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:59:23.038192 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:59:23.038476 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:59:23.038928 18506 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 08:59:27.366971 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:27.367472 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:59:32.369964 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:32.370273 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:59:32.539901 18641 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:59:32.540409 18641 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:59:32.542438 18641 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 08:59:32.544013 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 08:59:32.544531 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:59:32.544766 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 08:59:32.545275 18641 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:59:37.373751 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:37.374285 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 08:59:37.589493 17341 heartbeater.cc:646] Failed to heartbeat to 127.13.146.60:34701 (94 consecutive failures): Network error: Failed to ping master at 127.13.146.60:34701: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 08:59:37.932826 17185 heartbeater.cc:646] Failed to heartbeat to 127.13.146.62:43161 (118 consecutive failures): Network error: Failed to ping master at 127.13.146.62:43161: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:59:38.092254 17494 heartbeater.cc:646] Failed to heartbeat to 127.13.146.60:34701 (95 consecutive failures): Network error: Failed to ping master at 127.13.146.60:34701: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:59:41.272001 18729 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:59:41.272555 18729 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:59:41.274618 18729 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:59:41.276679 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 08:59:41.279445 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:59:41.279984 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:59:41.280301 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:59:41.280901 18729 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 08:59:42.377254 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:42.377707 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:59:44.136152 18751 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 08:59:44.136533 18751 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 08:59:44.138088 18751 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 08:59:44.143180 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 08:59:44.143615 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 08:59:44.143906 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 08:59:44.144440 18751 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 08:59:47.381078 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:47.381522 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 08:59:51.624701 18799 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 08:59:51.625197 18799 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 08:59:51.627269 18799 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 08:59:51.628916 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 08:59:51.629408 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 08:59:51.629637 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 08:59:51.630101 18799 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 08:59:52.384460 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:52.384783 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 08:59:54.340195 17179 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6ac0 after lost signal to thread 17049
W20251211 08:59:54.341156 17179 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8960 after lost signal to thread 17182
W20251211 08:59:56.744771 17337 debug-util.cc:398] Leaking SignalData structure 0x7b08000d6260 after lost signal to thread 17194
W20251211 08:59:56.745688 17337 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0c60 after lost signal to thread 17340
I20251211 08:59:57.388335 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 08:59:57.388919 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:00:00.428975 18851 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:00:00.429445 18851 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:00:00.430941 18851 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:00:00.433053 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:00:00.434623 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:00:00.434926 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:00:00.435544 18851 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 09:00:02.324323 17490 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5520 after lost signal to thread 17352
W20251211 09:00:02.325212 17490 debug-util.cc:398] Leaking SignalData structure 0x7b08000d1f00 after lost signal to thread 17493
I20251211 09:00:02.336206 18870 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:00:02.336695 18870 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:00:02.338624 18870 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:00:02.341269 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251211 09:00:02.346370 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:00:02.346904 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 09:00:02.347229 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:00:02.347702 18870 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:00:02.391041 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:02.391463 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 09:00:05.678051 17958 debug-util.cc:398] Leaking SignalData structure 0x7b08000ab720 after lost signal to thread 17865
W20251211 09:00:05.679176 17958 debug-util.cc:398] Leaking SignalData structure 0x7b080006f720 after lost signal to thread 17961
I20251211 09:00:07.411893 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:07.412302 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:00:12.414803 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:12.415134 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:00:14.491441 19037 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:00:14.491945 19037 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:00:14.493891 19037 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:00:14.495450 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:00:14.495896 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:00:14.496131 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:00:14.496606 19037 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:00:14.933809 19045 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:00:14.934221 19045 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:00:14.935688 19045 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:00:14.937430 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 09:00:14.940387 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:00:14.940915 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:00:14.941224 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:00:14.941754 19045 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:00:17.418598 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:17.419189 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:00:19.843762 19121 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:00:19.844295 19121 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:00:19.845806 19121 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:00:19.850590 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:00:19.851140 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:00:19.851456 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:00:19.852013 19121 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:00:22.422235 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:22.422672 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:00:26.356351 19206 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:00:26.356830 19206 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:00:26.358439 19206 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:00:26.361088 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251211 09:00:26.364244 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:00:26.364665 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:00:26.364950 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:00:26.365507 19206 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:00:27.425791 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:27.426311 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:00:27.742946 19224 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:00:27.743462 19224 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:00:27.745146 19224 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:00:27.746528 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:00:27.747146 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:00:27.747489 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:00:27.748409 19224 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:00:31.309376 19275 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:00:31.309782 19275 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:00:31.311839 19275 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:00:31.313231 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:00:31.313671 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:00:31.313897 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:00:31.314392 19275 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 09:00:31.924921 17337 debug-util.cc:398] Leaking SignalData structure 0x7b08000db9c0 after lost signal to thread 17194
W20251211 09:00:31.925768 17337 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5ec0 after lost signal to thread 17340
I20251211 09:00:32.342883 19289 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:00:32.343217 19289 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:00:32.344635 19289 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:00:32.346357 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 09:00:32.349098 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:00:32.349489 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:00:32.349826 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:00:32.350328 19289 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:00:32.428361 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:32.428685 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:00:37.468003 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:37.468372 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 09:00:37.728076 17958 debug-util.cc:398] Leaking SignalData structure 0x7b080009ae20 after lost signal to thread 17865
W20251211 09:00:37.728600 17958 debug-util.cc:398] Leaking SignalData structure 0x7b08000ab180 after lost signal to thread 17961
W20251211 09:00:37.909582 17341 heartbeater.cc:646] Failed to heartbeat to 127.13.146.60:34701 (214 consecutive failures): Network error: Failed to ping master at 127.13.146.60:34701: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:00:38.311349 17183 heartbeater.cc:646] Failed to heartbeat to 127.13.146.60:34701 (215 consecutive failures): Network error: Failed to ping master at 127.13.146.60:34701: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:00:38.401674 17496 heartbeater.cc:646] Failed to heartbeat to 127.13.146.62:43161 (238 consecutive failures): Network error: Failed to ping master at 127.13.146.62:43161: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 09:00:42.470860 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:42.471180 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:00:45.812438 19481 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:00:45.812886 19481 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:00:45.814395 19481 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:00:45.815909 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:00:45.816382 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:00:45.816601 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:00:45.817095 19481 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:00:47.325948 17916 maintenance_manager.cc:419] P c3112fc6f4ae4541a2af2405a97c707b: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033356
I20251211 09:00:47.402318 17915 maintenance_manager.cc:643] P c3112fc6f4ae4541a2af2405a97c707b: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.075s user 0.041s sys 0.009s Metrics: {"bytes_written":11449,"cfile_init":1,"drs_written":1,"lbm_read_time_us":212,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1991,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":19,"thread_start_us":492,"threads_started":1,"wal-append.queue_time_us":749}
I20251211 09:00:47.404373 17916 maintenance_manager.cc:419] P c3112fc6f4ae4541a2af2405a97c707b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1193 bytes on disk
I20251211 09:00:47.405829 17915 maintenance_manager.cc:643] P c3112fc6f4ae4541a2af2405a97c707b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":220,"lbm_reads_lt_1ms":4}
I20251211 09:00:47.474372 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:47.474746 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:00:50.489423 19540 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:00:50.489890 19540 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:00:50.491806 19540 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:00:50.495054 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 09:00:50.498472 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:00:50.498884 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 09:00:50.499135 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:00:50.499595 19540 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:00:52.477334 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:52.477710 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:00:57.480893 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:00:57.481274 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:00:58.978518 19663 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:00:58.979158 19663 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:00:58.981460 19663 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:00:58.983036 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:00:58.983592 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:00:58.983901 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:00:58.984496 19663 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 09:01:01.857358 17490 debug-util.cc:398] Leaking SignalData structure 0x7b08000dc940 after lost signal to thread 17352
W20251211 09:01:01.858314 17490 debug-util.cc:398] Leaking SignalData structure 0x7b08000cffc0 after lost signal to thread 17493
I20251211 09:01:02.484047 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:02.484457 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:01:02.898943 19732 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:01:02.899401 19732 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:01:02.900930 19732 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:01:02.902068 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 09:01:02.905372 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:01:02.905824 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:01:02.906122 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:01:02.906673 19732 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:01:07.488224 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:07.488626 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:01:12.491364 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:12.491679 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:01:13.608328 19915 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:01:13.608891 19915 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:01:13.610651 19915 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:01:13.612741 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 09:01:13.615105 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:01:13.615727 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:01:13.616012 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:01:13.616609 19915 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:01:14.675866 19942 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:01:14.676378 19942 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:01:14.678365 19942 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:01:14.680022 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:01:14.680567 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:01:14.680811 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:01:14.681310 19942 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:01:17.495283 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:17.495819 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:01:19.092430 20018 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:01:19.092936 20018 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:01:19.094602 20018 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:01:19.096011 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:01:19.096525 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:01:19.096801 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:01:19.097316 20018 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251211 09:01:20.528138 17337 debug-util.cc:398] Leaking SignalData structure 0x7b08000df780 after lost signal to thread 17194
W20251211 09:01:20.528987 17337 debug-util.cc:398] Leaking SignalData structure 0x7b080006e5c0 after lost signal to thread 17340
I20251211 09:01:22.498315 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:22.498662 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:01:24.324787 20116 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:01:24.325253 20116 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:01:24.326946 20116 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:01:24.329033 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 09:01:24.331653 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:01:24.332039 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 09:01:24.332329 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:01:24.332824 20116 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:01:25.057497 20131 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:01:25.057946 20131 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:01:25.059424 20131 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:01:25.063809 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:01:25.064256 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:01:25.064561 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:01:25.065109 20131 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:01:27.501904 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:27.502347 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 09:01:29.429461 17179 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfbe0 after lost signal to thread 17049
W20251211 09:01:29.430256 17179 debug-util.cc:398] Leaking SignalData structure 0x7b08000ddd80 after lost signal to thread 17182
I20251211 09:01:32.332855 20226 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:01:32.333374 20226 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:01:32.335486 20226 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:01:32.336838 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:01:32.337261 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:01:32.337512 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:01:32.337960 20226 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:01:32.505177 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:32.505517 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:01:36.594810 20266 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:01:36.595368 20266 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:01:36.596977 20266 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:01:36.599171 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251211 09:01:36.601814 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:01:36.602444 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:01:36.602715 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:01:36.603364 20266 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:01:37.508285 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:37.508683 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 09:01:38.300452 17343 heartbeater.cc:646] Failed to heartbeat to 127.13.146.62:43161 (357 consecutive failures): Network error: Failed to ping master at 127.13.146.62:43161: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:01:38.636202 17183 heartbeater.cc:646] Failed to heartbeat to 127.13.146.60:34701 (335 consecutive failures): Network error: Failed to ping master at 127.13.146.60:34701: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:01:38.774333 17496 heartbeater.cc:646] Failed to heartbeat to 127.13.146.62:43161 (358 consecutive failures): Network error: Failed to ping master at 127.13.146.62:43161: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 09:01:42.511058 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:42.511427 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:01:47.514343 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:47.514719 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:01:48.596104 20435 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:01:48.596679 20435 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:01:48.598836 20435 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:01:48.600267 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:01:48.600744 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:01:48.601020 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:01:48.601593 20435 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:01:52.517145 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:52.517472 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:01:55.841475 20559 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:01:55.842012 20559 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:01:55.844267 20559 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:01:55.846630 17913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 09:01:55.849397 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:01:55.849804 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:01:55.850078 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:01:55.850562 20559 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:01:56.484508 20578 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:01:56.484922 20578 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:01:56.486474 20578 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:01:56.490832 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:01:56.491245 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:01:56.491531 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:01:56.491997 20578 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:01:57.520255 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:01:57.520642 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:01:59.120433 20625 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:01:59.120929 20625 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:01:59.122406 20625 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:01:59.123984 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:01:59.124403 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:01:59.124601 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:01:59.125041 20625 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:02:02.523264 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:02.523631 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:02:07.527081 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:07.527496 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:02:12.555961 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:12.556358 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 09:02:13.064404 17958 debug-util.cc:398] Leaking SignalData structure 0x7b080006fe60 after lost signal to thread 17865
W20251211 09:02:13.064853 17958 debug-util.cc:398] Leaking SignalData structure 0x7b080009b400 after lost signal to thread 17961
I20251211 09:02:13.542404 20862 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:02:13.542879 20862 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:02:13.544835 20862 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:02:13.548455 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251211 09:02:13.551460 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:02:13.551903 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 09:02:13.552201 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:02:13.552644 20862 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:02:17.560122 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:17.560657 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 09:02:17.595789 17490 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1100 after lost signal to thread 17352
W20251211 09:02:17.596711 17490 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5b40 after lost signal to thread 17493
I20251211 09:02:17.822104 20924 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:02:17.822643 20924 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:02:17.824795 20924 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:02:17.826309 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:02:17.826754 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:02:17.827011 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:02:17.827508 20924 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:02:22.578619 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:22.578986 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:02:22.632421 20985 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:02:22.632900 20985 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:02:22.634393 20985 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:02:22.635689 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 09:02:22.638195 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:02:22.638701 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:02:22.638976 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:02:22.639437 20985 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251211 09:02:23.413981 17958 debug-util.cc:398] Leaking SignalData structure 0x7b080006f780 after lost signal to thread 17865
W20251211 09:02:23.414657 17958 debug-util.cc:398] Leaking SignalData structure 0x7b080006fdc0 after lost signal to thread 17961
I20251211 09:02:24.748052 21014 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:02:24.748409 21014 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:02:24.749738 21014 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:02:24.754261 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:02:24.754688 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 09:02:24.754989 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:02:24.755448 21014 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:02:27.583154 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:27.583573 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:02:30.962455 21109 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:02:30.963066 21109 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:02:30.965104 21109 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:02:30.967864 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251211 09:02:30.970897 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:02:30.971279 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 09:02:30.971542 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:02:30.972018 21109 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:02:32.585932 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:32.586269 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:02:33.663439 21143 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:02:33.663975 21143 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:02:33.666400 21143 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:02:33.668005 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:02:33.668500 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:02:33.668725 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:02:33.669265 21143 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:02:37.589499 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:37.590003 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 09:02:38.650149 17341 heartbeater.cc:646] Failed to heartbeat to 127.13.146.60:34701 (454 consecutive failures): Network error: Failed to ping master at 127.13.146.60:34701: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:02:38.982388 17183 heartbeater.cc:646] Failed to heartbeat to 127.13.146.60:34701 (455 consecutive failures): Network error: Failed to ping master at 127.13.146.60:34701: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:02:39.135879 17494 heartbeater.cc:646] Failed to heartbeat to 127.13.146.60:34701 (455 consecutive failures): Network error: Failed to ping master at 127.13.146.60:34701: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:02:40.748373 21260 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:02:40.748844 21260 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:02:40.750448 21260 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:02:40.752023 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:02:40.752508 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:02:40.752741 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:02:40.753269 21260 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:02:42.592450 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:42.592844 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:02:45.946520 21336 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:02:45.947151 21336 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:02:45.948700 21336 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:02:45.951356 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 09:02:45.954288 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:02:45.954841 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:02:45.955139 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:02:45.955650 21336 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:02:47.595649 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:47.596057 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:02:49.982029 21409 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:02:49.982380 21409 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:02:49.983809 21409 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:02:49.988003 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:02:49.988615 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:02:49.988840 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:02:49.989252 21409 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251211 09:02:52.531114 17337 debug-util.cc:398] Leaking SignalData structure 0x7b08000c21e0 after lost signal to thread 17194
W20251211 09:02:52.532042 17337 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8280 after lost signal to thread 17340
I20251211 09:02:52.598750 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:52.599155 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:02:57.602206 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:02:57.602645 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:03:00.598117 21586 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:03:00.598636 21586 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:03:00.600663 21586 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:03:00.602387 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:03:00.602950 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:03:00.603194 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:03:00.603760 21586 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:03:02.605216 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:03:02.605532 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:03:06.497219 21685 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:03:06.497695 21685 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:03:06.499702 21685 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:03:06.501837 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251211 09:03:06.504217 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:03:06.504580 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 09:03:06.504902 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:03:06.505352 21685 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:03:07.608225 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:03:07.608595 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:03:10.834807 21742 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:03:10.835590 21742 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:03:10.839259 21742 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:03:10.840188 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:03:10.840889 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:03:10.841120 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:03:10.841677 21742 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:03:12.611513 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:03:12.611953 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:03:17.043470 21825 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:03:17.044003 21825 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:03:17.045572 21825 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:03:17.048475 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 09:03:17.051105 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:03:17.051507 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
I20251211 09:03:17.051808 17911 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:03:17.052275 21825 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:03:17.614851 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:03:17.615348 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:03:22.618407 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:03:22.618808 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:03:23.331204 21911 raft_consensus.cc:493] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251211 09:03:23.331761 21911 raft_consensus.cc:515] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467cf4890bc486bbfb9b8f4bea2c2cf" member_type: VOTER last_known_addr { host: "127.13.146.3" port: 41219 } } peers { permanent_uuid: "62329f65aae54e84b396a51dd81f1b70" member_type: VOTER last_known_addr { host: "127.13.146.1" port: 46231 } } peers { permanent_uuid: "e8a2e7dba69c4c80bb3672eeaf63e1c2" member_type: VOTER last_known_addr { host: "127.13.146.2" port: 40951 } }
I20251211 09:03:23.333679 21911 leader_election.cc:290] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231), e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951)
W20251211 09:03:23.335294 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62329f65aae54e84b396a51dd81f1b70 (127.13.146.1:46231): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
W20251211 09:03:23.335788 17382 leader_election.cc:343] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8a2e7dba69c4c80bb3672eeaf63e1c2 (127.13.146.2:40951): Not found: Tablet not found: 128b128c966b4dcfa6241b96e3ae61b1
I20251211 09:03:23.336040 17382 leader_election.cc:304] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7467cf4890bc486bbfb9b8f4bea2c2cf; no voters: 62329f65aae54e84b396a51dd81f1b70, e8a2e7dba69c4c80bb3672eeaf63e1c2
I20251211 09:03:23.336535 21911 raft_consensus.cc:2749] T 128b128c966b4dcfa6241b96e3ae61b1 P 7467cf4890bc486bbfb9b8f4bea2c2cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251211 09:03:27.622747 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:03:27.623358 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 09:03:29.249143 17179 debug-util.cc:398] Leaking SignalData structure 0x7b08000b74a0 after lost signal to thread 17049
W20251211 09:03:29.250146 17179 debug-util.cc:398] Leaking SignalData structure 0x7b08000b93a0 after lost signal to thread 17182
I20251211 09:03:30.578454 22033 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b8dd1b89d5ad4e529459606ce6f85ab8)
I20251211 09:03:30.578963 22033 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133269af8ce44b579ebbb7805bc58800" member_type: VOTER last_known_addr { host: "127.13.146.62" port: 43161 } } peers { permanent_uuid: "c3112fc6f4ae4541a2af2405a97c707b" member_type: VOTER last_known_addr { host: "127.13.146.61" port: 36035 } } peers { permanent_uuid: "b8dd1b89d5ad4e529459606ce6f85ab8" member_type: VOTER last_known_addr { host: "127.13.146.60" port: 34701 } }
I20251211 09:03:30.580709 22033 leader_election.cc:290] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161), b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701)
W20251211 09:03:30.582356 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251211 09:03:30.584920 17911 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 133269af8ce44b579ebbb7805bc58800 (127.13.146.62:43161): Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:03:30.585345 17913 leader_election.cc:336] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b8dd1b89d5ad4e529459606ce6f85ab8 (127.13.146.60:34701): Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
I20251211 09:03:30.585593 17913 leader_election.cc:304] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3112fc6f4ae4541a2af2405a97c707b; no voters: 133269af8ce44b579ebbb7805bc58800, b8dd1b89d5ad4e529459606ce6f85ab8
I20251211 09:03:30.586072 22033 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c3112fc6f4ae4541a2af2405a97c707b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251211 09:03:32.626706 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:03:32.627084 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251211 09:03:37.630539 17857 master_runner.cc:186] Error getting master registration for 127.13.146.62:43161: Network error: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111),
I20251211 09:03:37.630998 17857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251211 09:03:39.006978 17343 heartbeater.cc:646] Failed to heartbeat to 127.13.146.62:43161 (597 consecutive failures): Network error: Failed to ping master at 127.13.146.62:43161: Client connection negotiation failed: client connection to 127.13.146.62:43161: connect: Connection refused (error 111)
W20251211 09:03:39.066601 17504 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701, num_attempts: 753) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:03:39.067205 17504 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701, num_attempts: 753) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
W20251211 09:03:39.067471 17504 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701, num_attempts: 753) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
F20251211 09:03:39.081498 17518 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701, num_attempts: 753) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251211 09:03:39.083566 17513 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_097cc85012564d15aa8db749266d3068 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.13.146.62:43161,127.13.146.61:36035,127.13.146.60:34701, num_attempts: 753) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.13.146.60:34701: connect: Connection refused (error 111)
*** Aborted at 1765443819 (unix time) try "date -d @1765443819" if you are using GNU date ***
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003648) received by PID 13896 (TID 0x7f08766f8700) from PID 13896; stack trace: ***
@ 0x2de460 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e0943 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f088afa0980 (unknown) at ??:0
@ 0x7f088614dfb7 gsignal at ??:0
@ 0x7f088614f921 abort at ??:0
@ 0x2e4456 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f08891e5da4 google::LogMessage::SendToLog() at ??:0
@ 0x7f08891e6950 google::LogMessage::Flush() at ??:0
@ 0x7f08891eba8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x360dae kudu::MasterStressTest::DeleteTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38e45e _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846
@ 0x38e311 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38da5b std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2de320 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7f088af956db start_thread at ??:0
@ 0x7f088623071f clone at ??:0